Voice cloning runs through a more rigorous process that begins with acquiring large quantity of audio data they can get from the target speaker. This is ideally at least 30 to 60 minutes of high-quality, isolated speech for best results. Those more modern models, like the ones produced by companies like Descript, can allegedly easily get a good impression from only seconds worth of their dataset (though this will be less precise and contain fewer emotional layers).
The underlying technology that has gained popularity, voice cloning is based on neural networks and more specifically deep learning models. Speech patterns are generated using Recurrent Neural Networks (RNNs), or Transformer-based architectures like GPT by algorithms like Text-to-Speech (TTS) systems. The models are trained to sound like the speaker in tone, pitch and cadence. With the help of GAN Voice generation systems like Google's WaveNet have become more human-like. The approach leads to 10% enhancement in audio realism over normal TTS methods.
The training of these models would take hours or even days depending on the complexity and dataset size. Some models, like those that are run on a GPU cluster and may take 48 hours to train (~$1k+ in computational $). Some of our customers are spending over $50,000 on their cloud bill every year just to reduce the time taken for training.
Applications for the real-time voice cloning like virtual assistants (think voice controlled car) are only viable if the inference process is done quickly and efficiently, The voice models latency should be below 100 milliseconds to provide a smooth user experience. For example, VocaliD uses voice cloning in the healthcare industry to give people suffering from speech impairments a means of communication by offering an individualized and unique voice in just a tiny fraction of the time it took for traditional voice banking.
The most striking has been the improvement of speaker adaptation in voice cloning, which fine-tunes a pre-trained voice model by providing only a few seconds of audio from a new speaker. This enables instant personalization coupled with a 90% homophonic property. This system lets brands build unique voice avatars for customer service or content production on a mass scale that have increased some engagement by 15 percent.
That applies to voice cloning as well: Elon Musk has warned us in the past that AI development should be “watched very closely” for fear of abuse. Lyrebird and Adobe, for instance, reveal the potential for perfectly replicable sound-alikes that worry concepts of privacy and consent. These risks underscore the importance of regulation to guard against any industry, like media and law enforcement, abusing its new powers.
DupDub offers voice cloning: the process and applications of creating high quality and lifelike voice clones using advanced technology.