Deep learning speech synthesis

Short description: Method of speech synthesis that uses deep neural networks

Deep learning speech synthesis refers to the application of deep learning models to generate natural-sounding human speech from written text (text-to-speech) or spectrum (vocoder). Deep neural networks are trained using large amounts of recorded speech and, in the case of a text-to-speech system, the associated labels and/or input text.

## Contents

* 1 Formulation
* 2 History
* 3 Semi-supervised learning
* 4 Zero-shot speaker adaptation
* 5 Neural vocoder
* 6 References

## Formulation

Given an input text or some sequence of linguistic units Y, the target speech X can be derived by

X=arg⁡max⁡P(X|Y,θ)

where θ is the set of model parameters.

Typically, the input text will first be passed to an acoustic feature generator, then the acoustic features are passed to the neural vocoder. For the acoustic feature generator, the loss function is typically L1 loss (Mean Absolute Error, MAE) or L2 loss (Mean Square Error, MSE). These loss functions impose a constraint that the output acoustic feature distributions must be Gaussian or Laplacian. In practice, since the human voice band ranges from approximately 300 to 4000 Hz, the loss function will be designed to have more penalty on this range:

loss=αlosshuman+(1−α)lossother

where losshuman is the loss from human voice band and α is a scalar, typically around 0.5. The acoustic feature is typically a spectrogram or Mel scale. These features capture the time-frequency relation of the speech signal, and thus are sufficient to generate intelligent outputs. The Mel-frequency cepstrum feature used in the speech recognition task is not suitable for speech synthesis, as it reduces too much information.

## History

For broader coverage of this topic, see History of speech synthesis.

A stack of dilated causal convolutional layers used in WaveNet[1]

In September 2016, DeepMind released WaveNet, which demonstrated that deep learning-based models are capable of modeling raw waveforms and generating speech from acoustic features like spectrograms or mel-spectrograms. Although WaveNet was initially considered to be computationally expensive and slow to be used in consumer products at the time, a year after its release, DeepMind unveiled a modified version of WaveNet known as "Parallel WaveNet," a production model 1,000 faster than the original.[1]

A comparison of the alignments (attentions) between Tacotron and a modified variant of Tacotron

This was followed by Google AI's Tacotron 2 in 2018, which demonstrated that neural networks could produce highly natural speech synthesis but required substantial training data—typically tens of hours of audio—to achieve acceptable quality. Tacotron 2 used an autoencoder architecture with attention mechanisms to convert input text into mel-spectrograms, which were then converted to waveforms using a separate neural vocoder. When trained on smaller datasets, such as 2 hours of speech, the output quality degraded while still being able to maintain intelligible speech, and with just 24 minutes of training data, Tacotron 2 failed to produce intelligible speech.[2]

In 2019, Microsoft Research introduced FastSpeech, which addressed speed limitations in autoregressive models like Tacotron 2.[3] FastSpeech utilized a non-autoregressive architecture that enabled parallel sequence generation, significantly reducing inference time while maintaining audio quality. Its feedforward transformer network with length regulation allowed for one-shot prediction of the full mel-spectrogram sequence, avoiding the sequential dependencies that bottlenecked previous approaches.[3] The same year saw the release of HiFi-GAN, a generative adversarial network (GAN)-based vocoder that improved the efficiency of waveform generation while producing high-fidelity speech.[4] In 2020, the release of Glow-TTS introduced a flow-based approach that allowed for fast inference and voice style transfer capabilities.[5]

In March 2020, the free text-to-speech website 15.ai was launched. 15.ai gained widespread international attention in early 2021 for its ability to synthesize emotionally expressive speech of fictional characters from popular media with minimal amount of data.[6][7][8] The creator of 15.ai (known pseudonymously as 15) stated that 15 seconds of training data is sufficient to perfectly clone a person's voice (hence its name, "15.ai"), a significant reduction from the previously known data requirement of tens of hours.[9] 15.ai is credited as the first platform to popularize AI voice cloning in memes and content creation.[10][11][9] 15.ai used a multi-speaker model that enabled simultaneous training of multiple voices and emotions, implemented sentiment analysis using DeepMoji, and supported precise pronunciation control via ARPABET.[9][6] The 15-second data efficiency benchmark was later corroborated by OpenAI in 2024.[12]

## Semi-supervised learning

Currently, self-supervised learning has gained much attention through better use of unlabelled data. Research has shown that, with the aid of self-supervised loss, the need for paired data decreases.[13][14]

## Zero-shot speaker adaptation

Zero-shot speaker adaptation is promising because a single model can generate speech with various speaker styles and characteristic. In June 2018, Google proposed to use pre-trained speaker verification models as speaker encoders to extract speaker embeddings.[15] The speaker encoders then become part of the neural text-to-speech models, so that it can determine the style and characteristics of the output speech. This procedure has shown the community that it is possible to use only a single model to generate speech with multiple styles.

## Neural vocoder

File:Larynx-HiFi-GAN speech sample.wav In deep learning-based speech synthesis, neural vocoders play an important role in generating high-quality speech from acoustic features. The WaveNet model proposed in 2016 achieves excellent performance on speech quality. Wavenet factorised the joint probability of a waveform 𝐱={x1,...,xT} as a product of conditional probabilities as follows

pθ(𝐱)=∏t=1Tp(xt|x1,...,xt−1)

where θ is the model parameter including many dilated convolution layers. Thus, each audio sample xt is conditioned on the samples at all previous timesteps. However, the auto-regressive nature of WaveNet makes the inference process dramatically slow. To solve this problem, Parallel WaveNet[16] was proposed. Parallel WaveNet is an inverse autoregressive flow-based model which is trained by knowledge distillation with a pre-trained teacher WaveNet model. Since such inverse autoregressive flow-based models are non-auto-regressive when performing inference, the inference speed is faster than real-time. Meanwhile, Nvidia proposed a flow-based WaveGlow[17] model, which can also generate speech faster than real-time. However, despite the high inference speed, parallel WaveNet has the limitation of needing a pre-trained WaveNet model, so that WaveGlow takes many weeks to converge with limited computing devices. This issue has been solved by Parallel WaveGAN,[18] which learns to produce speech through multi-resolution spectral loss and GAN learning strategies.

Synthesis example

File:The Chaos synthesized by VITS.ogg

The Chaos (short version) synthesized by VITS, a research deep-learning-based end-to-end text-to-speech method, using the LJ Speech dataset.

|

* * *

Problems playing this file? See media help.

## References

1. ↑ 1.0 1.1 van den Oord, Aäron (2017-11-12). "High-fidelity speech synthesis with WaveNet". https://www.deepmind.com/blog/high-fidelity-speech-synthesis-with-wavenet.
2. ↑ "Audio samples from "Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis"". 2018-08-30. https://google.github.io/tacotron/publications/semisupervised/index.html.
3. ↑ 3.0 3.1 Ren, Yi (2019). "FastSpeech: Fast, Robust and Controllable Text to Speech". arXiv:1905.09263 [cs.CL].
4. ↑ Kong, Jungil (2020). "HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis". arXiv:2010.05646 [cs.SD].
5. ↑ Kim, Jaehyeon (2020). "Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search". arXiv:2005.11129 [eess.AS].
6. ↑ 6.0 6.1 Kurosawa, Yuki (January 19, 2021). "ゲームキャラ音声読み上げソフト「15.ai」公開中。『Undertale』や『Portal』のキャラに好きなセリフを言ってもらえる" (in ja). https://automaton-media.com/articles/newsjp/20210119-149494/.
7. ↑ 遊戲, 遊戲角落 (January 20, 2021). "這個AI語音可以模仿《傳送門》GLaDOS講出任何對白！連《Undertale》都可以學" (in zh-tw). https://game.udn.com/game/story/10453/5189551.
8. ↑ Lamorlette, Robin (January 25, 2021). "Insolite : un site permet de faire dire ce que vous souhaitez à GlaDOS (et à d'autres personnages de jeux vidéo)" (in fr). https://www.clubic.com/technologies-d-avenir/intelligence-artificielle/actualite-349691-insolite-un-site-permet-de-faire-dire-ce-que-vous-souhaitez-a-glados-et-a-d-autres-personnages-de-jeux-video-.html.
9. ↑ 9.0 9.1 9.2 Temitope, Yusuf (December 10, 2024). "15.ai Creator reveals journey from MIT Project to internet phenomenon". The Guardian (Lagos, Nigeria). https://guardian.ng/technology/15-ai-creator-reveals-journey-from-mit-project-to-internet-phenomenon/.
10. ↑ Anirudh VK (March 18, 2023). "Deepfakes Are Elevating Meme Culture, But At What Cost?". https://analyticsindiamag.com/ai-origins-evolution/deepfakes-are-elevating-meme-culture-but-at-what-cost/.
11. ↑ Wright, Steven (March 21, 2023). "Why Biden, Trump, and Obama Arguing Over Video Games Is YouTube's New Obsession". https://www.inverse.com/gaming/youtube-ai-presidential-gaming-debates.
12. ↑ "Navigating the Challenges and Opportunities of Synthetic Voices". March 9, 2024. https://openai.com/index/navigating-the-challenges-and-opportunities-of-synthetic-voices/.
13. ↑ Chung, Yu-An (2018). "Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis". arXiv:1808.10128 [cs.CL].
14. ↑ Ren, Yi (2019). "Almost Unsupervised Text to Speech and Automatic Speech Recognition". arXiv:1905.06791 [cs.CL].
15. ↑ Jia, Ye (2018). "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis". arXiv:1806.04558 [cs.CL].
16. ↑ van den Oord, Aaron (2018). "Parallel WaveNet: Fast High-Fidelity Speech Synthesis". arXiv:1711.10433 [cs.CL].
17. ↑ Prenger, Ryan (2018). "WaveGlow: A Flow-based Generative Network for Speech Synthesis". arXiv:1811.00002 [cs.SD].
18. ↑ Yamamoto, Ryuichi (2019). "Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram". arXiv:1910.11480 [eess.AS].

* v
* t
* e

Speech synthesis

Free software|

| Speaking|

* eSpeak]]/eSpeakNG
* Gnopernicus
* Gnuspeech
* Orca
* Festival Speech Synthesis System/Flite
* FreeTTS
* Automatik Text Reader

|

Singing|

* eCantorix
* Lyricos / Flinger
* Sinsy

Proprietary
software|

| Speaking|

* DECtalk
* Software Automatic Mouth
* Talk It!
* Microsoft Agent
* Microsoft Speech API
* Microsoft text-to-speech voices
* Readspeaker
* Voice browser
* CoolSpeech
* IVONA
* CereProc
* CeVIO Creative Studio
* Voiceroid
* LaLaVoice

|

Singing|

* Alter/Ego
* Cantor
* CeVIO Creative Studio
* Chipspeech
* Symphonic Choirs
* Vocaloid
* NIAONiao Virtual Singer
* Vocalina
* Utau
* Realivox
* PPG Phonem

Machine|

* Echo 2
* Mockingboard
* Pattern playback
* Phasor
* RIAS
* Texas Instruments LPC Speech Chips
* General Instrument SP0256
* TuVox

Applications|

* AOLbyPhone
* DialogOS
* Dr. Sbaitso
* MBROLA
* Microsoft Narrator
* Microsoft Speech Server
* PlainTalk
* Voice font

Protocols|

* Speech Synthesis Markup Language
* SABLE
* VoiceXML

Developers/
Researchers|

* Alan W. Black
* Catherine Browman
* Franklin Seaney Cooper
* Gunnar Fant
* Haskins Laboratories
* Wolfgang von Kempelen
* Ignatius Mattingly
* Philip Rubin
* Yamaha

Process|

* Articulatory synthesis
* Concatenative synthesis
* Currah
* Inverse filter
* PSOLA
* Phase vocoder
* Self-voicing

* v
* t
* e

Artificial intelligence (AI)

* History
* timeline
* Glossary
* Companies
* Projects

Concepts|

* Parameter
* Hyperparameter
* Loss functions
* Regression
* Bias–variance tradeoff
* Double descent
* Overfitting
* Clustering
* Gradient descent
* SGD
* Quasi-Newton method
* Conjugate gradient method
* Backpropagation
* Attention
* Convolution
* Normalization
* Batchnorm
* Activation
* Softmax
* Sigmoid
* Rectifier
* Gating
* Weight initialization
* Regularization
* Datasets
* Augmentation
* Prompt engineering
* Reinforcement learning
* Q-learning
* SARSA
* Imitation
* Policy gradient
* Diffusion
* Latent diffusion model
* Autoregression
* Adversary
* RAG
* Uncanny valley
* RLHF
* Self-supervised learning
* Reflection
* Recursive self-improvement
* Hallucination
* Word embedding
* Vibe coding

Applications|

* Machine learning
* In-context learning
* Artificial neural network
* Deep learning
* Language model
* Large
* NMT
* Reasoning
* Model Context Protocol
* Intelligent agent
* Artificial human companion
* Humanity's Last Exam
* Lethal autonomous weapons (LAWs)
* Generative artificial intelligence (GenAI)
* (Hypothetical: Artificial general intelligence (AGI))
* (Hypothetical: Artificial superintelligence (ASI))
* Agent2Agent protocol

Implementations|

| Audio–visual|

* AlexNet
* WaveNet
* Human image synthesis
* HWR
* OCR
* Computer vision
* Speech synthesis
* 15.ai
* ElevenLabs
* Speech recognition
* Whisper
* Facial recognition
* AlphaFold
* Text-to-image models
* Aurora
* DALL-E
* Firefly
* Flux
* GPT Image
* Ideogram
* Imagen
* Midjourney
* Recraft
* Stable Diffusion
* Text-to-video models
* Dream Machine
* Runway Gen
* Hailuo AI
* Kling
* Sora
* Seedance
* Veo
* Music generation
* Riffusion
* Suno AI
* Udio

|

Text|

* Word2vec
* Seq2seq
* GloVe
* BERT
* T5
* Llama
* Chinchilla AI
* PaLM
* GPT
* 1
* 2
* 3
* J
* ChatGPT
* 4
* 4o
* o1
* o3
* 4.5
* 4.1
* o4-mini
* 5
* 5.1
* 5.2
* Claude
* Gemini
* Gemini (language model)
* Gemma
* Grok
* LaMDA
* BLOOM
* DBRX
* Project Debater
* IBM Watson
* IBM Watsonx
* Granite
* PanGu-Σ
* DeepSeek
* Qwen

Decisional|

* AlphaGo
* AlphaZero
* OpenAI Five
* Self-driving car
* MuZero
* Action selection
* AutoGPT
* Robot control

People|

* Alan Turing
* Warren Sturgis McCulloch
* Walter Pitts
* John von Neumann
* Christopher D. Manning
* Claude Shannon
* Shun'ichi Amari
* Kunihiko Fukushima
* Takeo Kanade
* Marvin Minsky
* John McCarthy
* Nathaniel Rochester
* Allen Newell
* Cliff Shaw
* Herbert A. Simon
* Oliver Selfridge
* Frank Rosenblatt
* Bernard Widrow
* Joseph Weizenbaum
* Seymour Papert
* Seppo Linnainmaa
* Paul Werbos
* Geoffrey Hinton
* John Hopfield
* Jürgen Schmidhuber
* Yann LeCun
* Yoshua Bengio
* Lotfi A. Zadeh
* Stephen Grossberg
* Alex Graves
* James Goodnight
* Andrew Ng
* Fei-Fei Li
* Alex Krizhevsky
* Ilya Sutskever
* Oriol Vinyals
* Quoc V. Le
* Ian Goodfellow
* Demis Hassabis
* David Silver
* Andrej Karpathy
* Ashish Vaswani
* Noam Shazeer
* Aidan Gomez
* John Schulman
* Mustafa Suleyman
* Jan Leike
* Daniel Kokotajlo
* François Chollet

Architectures|

* Neural Turing machine
* Differentiable neural computer
* Transformer
* Vision transformer (ViT)
* Recurrent neural network (RNN)
* Long short-term memory (LSTM)
* Gated recurrent unit (GRU)
* Echo state network
* Multilayer perceptron (MLP)
* Convolutional neural network (CNN)
* Residual neural network (RNN)
* Highway network
* Mamba
* Autoencoder
* Variational autoencoder (VAE)
* Generative adversarial network (GAN)
* Graph neural network (GNN)

Political|

* AI safety (Alignment)
* Ethics of AI
* EU AI Act
* Precautionary principle
* Regulation of AI
* Virtual politician

Social and economic|

* AI boom
* AI bubble
* AI literacy
* AI slop
* AI veganism
* AI winter
* Anthropomorphism
* In architecture
* In education
* In healthcare
* Chatbot psychosis
* Mental health
* In visual art

* Category

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Deep learning speech synthesis. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Deep_learning_speech_synthesis&oldid=4347102"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template