Human or AI? I cloned my voice and asked friends if they could tell the difference

SINGAPORE: “Hi, how have you been? So I heard about this restaurant that just recently opened up. Want to go check it out the next time we meet?”

These were the innocuous sentences used in a simple experiment to find out if people in my social circle could distinguish my real voice from an AI-cloned version.

The result was some confusion - but more importantly, the ease with which the imitation was generated suggests that more attention should be paid to the phenomenon of deepfake voice phishing, or vishing.

Millions of dollars have been lost to scammers using cheap yet increasingly sophisticated artificial intelligence tools to impersonate the voices of real people.

In Asia-Pacific, the trend of AI and deepfake-enabled fraud is accelerating even faster than the rest of the world, according to cybersecurity firm Group-IB.

AI-related fraud attempts surged by 194 per cent in 2024 compared to 2023, with deepfake vishing emerging as one of the most commonly used methods, said the company's senior fraud analyst for the region Yuan Huang.

An increase has also been observed in Singapore. While exact vishing figures are not publicly available, Ms Huang pointed to a study which found that about 56 per cent of businesses here have experienced deepfake audio fraud.

The Cyber Security Agency of Singapore told CNA that audio deepfakes are a rising concern, due to how they exploit people's natural inclination to trust familiar voices, such as those seemingly from a family member or a colleague.

COPYING MY VOICE

My experiment, which was facilitated by Ms Huang from Group-IB at the start of August, showed that cloning a voice really did not take much: A publicly available online tool – of which there are several – and samples of my voice as short as 10 seconds.

A more convincing replication would require longer samples and more tone variations.

The subscription plan for the platform we chose was priced at less than US$10 for the first month. Elsewhere, fees can start at US$3 a month.

Under our plan, we could generate high-definition audio output and cloned voices in other languages.

Seconds after uploading my samples, my deepfaked voice was ready.

On the user-friendly interface, cloned voices can be fine-tuned by adjusting speeds or tweaking settings to be more monotonous or more dramatic. None of this required any particular technical skills; practically anyone can generate a cloned voice. 

When I finally played back my AI-generated voice in full, I was taken by surprise.

I already knew it was supposed to sound like me. But the level of similarity and accuracy – down to the pauses I habitually take in between words – was something else.

The technology was not entirely flawless. The voice clone seemed to have a hint of an American accent; and the more we replayed it, the less it sounded like me.

But it would suffice to fool most people, according to studies at least.

A 2024 poll by the University of California in the United States found that participants mistook the identity of an AI voice for its real counterpart 80 per cent of the time.

When researchers at the University College London played audio clips to 100 people, just 48 per cent were able to tell which was a human voice and which was created using AI.

A recent public awareness survey by CSA found that a majority (78 per cent) had confidence in their ability to identify deepfakes. But only one in four could distinguish legitimate videos from AI-manipulated ones. No data or trends specific to audio content were available.

In an April 2024 reply to a parliamentary question, it was also revealed that the Singapore police has not been tracking the number of deepfake-related scams using video, voice and other media.

Home Affairs Minister K Shanmugam said then: "While we have received some reports where the complainants had alleged that deepfake techniques were used by the scammers, the number is not high."

PUT TO THE TEST

After the cloning came the fun part of the experiment.

We typed out the statement on checking out a restaurant - something I would say to friends - on the online platform; then separately recorded myself saying the same thing.

Then I called four contacts - two colleagues as well as two friends I've known for almost a decade - and played them the two audio clips.

One of my colleagues initially mistook my real voice for AI. But in the end, all four could tell the difference: According to them, no amount of tweaking of settings could take away the more robotic and monotonous flavour of the cloned voice.

One possible factor could be that the Singaporean accent is not as easy to clone, since most AI models are trained on American or British accents, said Associate Professor Terence Sim from the National University of Singapore's (NUS) school of computing. 

My friends were also young adults in their 20s, who may be more aware of the hallmarks of AI use - and on higher alert in the environment of phone calls.

THE VISHING APPEAL

Experts CNA spoke to nonetheless noted the growing accessibility of low-cost and sometimes even free generative AI tools to convincingly mimic and create human voices.

A CSA spokesperson said that with audio deepfakes, "while there are traditional indicators like unnatural pauses in speech, robotic intonation, or unusual word choices, these tells are becoming harder to spot as AI technology advances".

"What makes audio deepfakes particularly challenging to detect is that humans are naturally attuned to trust voice communications, especially in urgent or emotional situations," he added.

"When someone appears to sound exactly like a family member or colleague in distress, the emotional response often overrides our usual caution."

The AI tools also allow scammers to spoof trusted institutions with accuracy, making it easier to manipulate targets and gain financial and sensitive information, said Assistant Professor Saifuddin Ahmed from the Wee Kim Wee School of Communication and Information at Nanyang Technological University (NTU).

"Combined with the availability of stolen personal data from breaches or user mistakes, the technology empowers scammers to craft highly convincing, personalised attacks."

Group-IB's Ms Huang also highlighted that social media is often the primary source for obtaining voice samples.

Other avenues include radio or TV interviews, webinars and public recordings available online.

"In some cases, scammers record a victim's voice during a phone call, particularly during impersonation or vishing attempts, to later use it for further attacks."

Scammers, being financially motivated, prefer targets who can potentially bring such high returns, Ms Huang said. This includes chief executives, chief financial officers and finance personnel. 

But they can also target the elderly, particularly those unfamiliar with AI-generated voice technology, or those who can be emotionally manipulated.

WHAT TO DO ABOUT IT

Attempting to regulate online voice cloning platforms would be a complex matter, with almost all of them hosted outside of Singapore's jurisdiction, experts told CNA.

Asst Prof Saifuddin from NTU said cooperation between governments and tech companies would be needed to set boundaries to safeguard users.

At an industry level, telecom providers can also implement advanced call authentication and improve their spam-blocking technologies, he said. 

Asst Prof Saifuddin's advice to individuals was to use a different communication channel where possible, such as text or email, to confirm the request. "Never rely solely on the incoming call for verification," he said.

NUS' Assoc Prof Sim pointed to skills that could be picked up, such as how to listen out for hissing, mismatched background sounds in different segments of the speech, a lack of background sound, or abrupt transitions from one sentence to the next.

But he also acknowledged that these were not foolproof, with voice cloning technology constantly improving.

CSA said to be on alert when receiving urgent and unsolicited requests for monetary transfers or sensitive details such as passwords and banking credentials.

"For voice calls from a supposed friend or family member, ask questions that only they would know ... If in doubt, do hang up and call the friend or family member directly on their known number," said the agency's spokesperson.

In the case of callers claiming to be from institutions, dial their official hotline to verify.

"A healthy dose of scepticism is important in today’s digital world," said CSA. "Not everything we see or hear online is what it appears to be, and if it’s too good to be true, then it probably is."

That was also my takeaway from the experiment, topped off with a dose of wariness. The next time I get a call from an unknown number, I won't be the first to speak.

Because all a scammer could need is a "hello, who's this" – and the next you know, an impersonator is calling your family and friends.