Deep learning speech synthesis

Short description: Method of speech synthesis that uses deep neural networks

Machine learning and
data mining

Problems

* Classification
* Clustering
* Regression
* Anomaly detection
* AutoML
* Association rules
* Reinforcement learning
* Structured prediction
* Feature engineering
* Feature learning
* Online learning
* Semi-supervised learning
* Unsupervised learning
* Learning to rank
* Grammar induction

Supervised learning
(classification • regression)

* Decision trees
* Ensembles
* Bagging
* Boosting
* Random forest
* k-NN
* Linear regression
* Naive Bayes
* Artificial neural networks
* Logistic regression
* Perceptron
* Relevance vector machine (RVM)
* Support vector machine (SVM)

Clustering

* BIRCH
* CURE
* Hierarchical
* k-means
* Expectation–maximization (EM)

DBSCAN
* OPTICS
* Mean-shift

Dimensionality reduction

* Factor analysis
* CCA
* ICA
* LDA
* NMF
* PCA
* t-SNE

Structured prediction

* Graphical models
* Bayes net
* Conditional random field
* Hidden Markov

Anomaly detection

* k-NN
* Local outlier factor

Artificial neural networks

* Autoencoder
* Deep learning
* DeepDream
* Multilayer perceptron
* RNN
* LSTM
* GRU
* Restricted Boltzmann machine
* GAN
* SOM
* Convolutional neural network
* U-Net

Reinforcement learning

* Q-learning
* SARSA
* Temporal difference (TD)

Theory

* Bias–variance dilemma
* Computational learning theory
* Empirical risk minimization
* Occam learning
* PAC learning
* Statistical learning
* VC theory

Machine-learning venues

* NeurIPS
* ICML
* ML
* JMLR
* ArXiv:cs.LG

Glossary of artificial intelligence

* Glossary of artificial intelligence

Related articles

* List of datasets for machine-learning research
* Outline of machine learning

* v
* t
* e

Deep learning speech synthesis uses Deep Neural Networks (DNN) to produce artificial speech from text (text-to-speech) or spectrum (vocoder). The deep neural networks are trained using a large amount of recorded speech and, in the case of a text-to-speech system, the associated labels and/or input text.

Some DNN-based speech synthesizers are approaching the naturalness of the human voice.

## Contents

* 1 Formulation
* 2 Brief history
* 3 Semi-supervised learning
* 4 Zero-shot speaker adaptation
* 5 Neural vocoder
* 6 References

## Formulation

Given an input text or some sequence of linguistic unit [math]\displaystyle{ Y }[/math], the target speech [math]\displaystyle{ X }[/math] can be derived by

[math]\displaystyle{ X=\arg\max P(X|Y, \theta) }[/math]

where [math]\displaystyle{ \theta }[/math] is the model parameter.

Typically, the input text will first be passed to an acoustic feature generator, then the acoustic features are passed to the neural vocoder. For the acoustic feature generator, the Loss function is typically L1 or L2 loss. These loss functions impose a constraint that the output acoustic feature distributions must be Gaussian or Laplacian. In practice, since the human voice band ranges from approximately 300 to 4000 Hz, the loss function will be designed to have more penalty on this range:

[math]\displaystyle{ loss=\alpha \text{loss}_{\text{human}} + (1 - \alpha) \text{loss}_{\text{other}} }[/math]

where [math]\displaystyle{ \text{loss}_{\text{human}} }[/math] is the loss from human voice band and [math]\displaystyle{ \alpha }[/math] is a scalar typically around 0.5. The acoustic feature is typically Spectrogram or spectrogram in Mel scale. These features capture the time-frequency relation of speech signal and thus, it is sufficient to generate intelligent outputs with these acoustic features. The Mel-frequency cepstrum feature used in the speech recognition task is not suitable for speech synthesis because it reduces too much information.

## Brief history

A stack of dilated casual convolutional layers used in WaveNet[1]

In September 2016, DeepMind proposed WaveNet, a deep generative model of raw audio waveforms, demonstrating that deep learning-based models are capable of modeling raw waveforms and generating speech from acoustic features like spectrograms or mel-spectrograms. Although WaveNet was initially considered to be computationally expensive and slow to be used in consumer products at the time, a year after its release, DeepMind unveiled a modified version of WaveNet known as "Parallel WaveNet," a production model 1,000 faster than the original.[1]

In early 2017, Mila proposed char2wav, a model to produce raw waveform in an end-to-end method. In the same year, Google and Facebook proposed Tacotron and VoiceLoop, respectively, to generate acoustic features directly from the input text; months later, Google proposed Tacotron2, which combined the WaveNet vocoder with the revised Tacotron architecture to perform end-to-end speech synthesis. Tacotron2 can generate high-quality speech approaching the human voice. Since then, end-to-end methods have become the hottest research topic because many researchers around the world have started to notice the power of end-to-end speech synthesizers.[2][3]

## Semi-supervised learning

Currently, self-supervised learning has gained much attention through better use of unlabelled data. Research[4][5] has shown that, with the aid of self-supervised loss, the need for paired data decreases.

## Zero-shot speaker adaptation

Zero-shot speaker adaptation is promising because a single model can generate speech with various speaker styles and characteristic. In June 2018, Google proposed to use pre-trained speaker verification models as speaker encoders to extract speaker embeddings.[6] The speaker encoders then become part of the neural text-to-speech models, so that it can determine the style and characteristics of the output speech. This procedure has shown the community that it is possible to use only a single model to generate speech with multiple styles.

## Neural vocoder

File:Larynx-HiFi-GAN speech sample.wav In deep learning-based speech synthesis, neural vocoders play an important role in generating high-quality speech from acoustic features. The WaveNet model proposed in 2016 achieves excellent performance on speech quality. Wavenet factorised the joint probability of a waveform [math]\displaystyle{ \mathbf{x}=\\{x_1,...,x_T\\} }[/math] as a product of conditional probabilities as follows

[math]\displaystyle{ p_{\theta}(\mathbf{x})=\prod_{t=1}^{T}p(x_t|x_1,...,x_{t-1}) }[/math]

where [math]\displaystyle{ \theta }[/math] is the model parameter including many dilated convolution layers. Thus, each audio sample [math]\displaystyle{ x_t }[/math] is conditioned on the samples at all previous timesteps. However, the auto-regressive nature of WaveNet makes the inference process dramatically slow. To solve this problem, Parallel WaveNet[7] was proposed. Parallel WaveNet is an inverse autoregressive flow-based model which is trained by knowledge distillation with a pre-trained teacher WaveNet model. Since such inverse autoregressive flow-based models are non-auto-regressive when performing inference, the inference speed is faster than real-time. Meanwhile, Nvidia proposed a flow-based WaveGlow[8] model, which can also generate speech faster than real-time. However, despite the high inference speed, parallel WaveNet has the limitation of needing a pre-trained WaveNet model, so that WaveGlow takes many weeks to converge with limited computing devices. This issue has been solved by Parallel WaveGAN,[9] which learns to produce speech through multi-resolution spectral loss and GAN learning strategies.

Synthesis example

File:The Chaos synthesized by VITS.ogg

The Chaos (short version) synthesized by VITS, a research deep-learning-based end-to-end text-to-speech method, using the LJ Speech dataset.

|

* * *

Problems playing this file? See media help.

## References

1. ↑ 1.0 1.1 van den Oord, Aäron (2017-11-12). "High-fidelity speech synthesis with WaveNet". https://www.deepmind.com/blog/high-fidelity-speech-synthesis-with-wavenet.
2. ↑ Hsu, Wei-Ning (2018). "Hierarchical Generative Modeling for Controllable Speech Synthesis". arXiv:1810.07217 [cs.CL].
3. ↑ Habib, Raza (2019). "Semi-Supervised Generative Modeling for Controllable Speech Synthesis". arXiv:1910.01709 [cs.CL].
4. ↑ Chung, Yu-An (2018). "Semi-Supervised Training for Improving Data Efficiency in End-to-End Speech Synthesis". arXiv:1808.10128 [cs.CL].
5. ↑ Ren, Yi (2019). "Almost Unsupervised Text to Speech and Automatic Speech Recognition". arXiv:1905.06791 [cs.CL].
6. ↑ Jia, Ye (2018). "Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis". arXiv:1806.04558 [cs.CL].
7. ↑ van den Oord, Aaron (2018). "Parallel WaveNet: Fast High-Fidelity Speech Synthesis". arXiv:1711.10433 [cs.CL].
8. ↑ Prenger, Ryan (2018). "WaveGlow: A Flow-based Generative Network for Speech Synthesis". arXiv:1811.00002 [cs.SD].
9. ↑ Yamamoto, Ryuichi (2019). "Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram". arXiv:1910.11480 [eess.AS].

* v
* t
* e

Differentiable computing

General|

* Differentiable programming
* Neural Turing machine
* Differentiable neural computer
* Automatic differentiation
* Neuromorphic engineering

Concepts|

* Gradient descent
* Cable theory
* Cluster analysis
* Regression analysis
* Pattern recognition
* Adversarial machine learning
* Computational learning theory

Programming languages|

* Python
* Julia

Application|

* Machine learning
* Artificial neural network
* Scientific computing
* Artificial Intelligence

Hardware|

* TPU
* VPU
* Memristor
* SpiNNaker

Software library|

* TensorFlow
* PyTorch

Implementation|

| Audio-visual|

* AlexNet
* WaveNet
* Human image synthesis
* HWR
* OCR
* Speech synthesis
* Speech recognition
* Facial recognition system

|

Verbal|

* Word2vec
* Transformer
* BERT
* NMT
* Project Debater
* Watson
* GPT-3

Decisional|

* AlphaGo
* Q-learning
* SARSA
* OpenAI Five

People|

* Alex Graves
* Ian Goodfellow
* Yoshua Bengio
* Geoffrey Hinton
* Yann LeCun
* Andrew Ng
* Demis Hassabis

* Portals
* Computer programming
* Technology
* Category
* Artificial neural networks
* Machine learning

* v
* t
* e

Speech synthesis

Free software|

| Speaking|

* eSpeak]]/eSpeakNG
* Gnopernicus
* Gnuspeech
* Orca
* Festival Speech Synthesis System/Flite
* FreeTTS
* Automatik Text Reader

|

Singing|

* eCantorix
* Lyricos / Flinger
* Sinsy

Proprietary
software|

| Speaking|

* DECtalk
* Software Automatic Mouth
* Talk It!
* Microsoft Agent
* Microsoft Speech API
* Microsoft text-to-speech voices
* Readspeaker
* Voice browser
* CoolSpeech
* IVONA
* CereProc
* CeVIO Creative Studio
* Voiceroid
* LaLaVoice

|

Singing|

* Alter/Ego
* Cantor
* CeVIO Creative Studio
* Chipspeech
* Symphonic Choirs
* Vocaloid
* NIAONiao Virtual Singer
* Vocalina
* Utau
* Realivox
* PPG Phonem

Machine|

* Echo 2
* Mockingboard
* Pattern playback
* Phasor
* RIAS
* Texas Instruments LPC Speech Chips
* General Instrument SP0256
* TuVox

Applications|

* AOLbyPhone
* DialogOS
* Dr. Sbaitso
* MBROLA
* Microsoft Narrator
* Microsoft Speech Server
* PlainTalk
* Voice font

Protocols|

* Speech Synthesis Markup Language
* SABLE
* VoiceXML

Developers/
Researchers|

* Alan W. Black
* Catherine Browman
* Franklin Seaney Cooper
* Gunnar Fant
* Haskins Laboratories
* Wolfgang von Kempelen
* Ignatius Mattingly
* Philip Rubin
* Yamaha

Process|

* Articulatory synthesis
* Concatenative synthesis
* Currah
* Inverse filter
* PSOLA
* Phase vocoder
* Self-voicing

0.00

(0 votes)

Original source: https://en.wikipedia.org/wiki/Deep learning speech synthesis. Read more

|

Retrieved from "https://handwiki.org/wiki/index.php?title=Deep_learning_speech_synthesis&oldid=3015909"
*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template