1. 6-Text To Speech (TTS) Speech Synthesis
Speech Synthesis Concept
Speech Naturalness
Phone Sequence To Speech
Articulatory Approaches
Concatenative Approaches
HMM-based Approaches
Rule-Based Approaches

2. Speech Synthesis Concept
Text
Speech waveform
Speech
waveform
Text
Text to
Phone Sequence
Phone Sequence
to Speech
Natural Language
Processing (NLP)
Speech Processing

3. Speech Naturalness
Obviation of undesirable noise and distortion and dissociation from speech
Prosody generation
Speech energy
Duration
pitch
Intonation
Stress

4. Speech Naturalness (Cont’d)
Intonation and Stress are very effective in speech naturalness
Intonation : Variation of Pitch frequency along speaking
Stress : Increasing the pitch frequency in a specific time

5. Which word receives an intonation?
It depends on the context.
The ‘new’ information in the answer to a question is often accented
while the ‘old’ information is usually not.
Q1: What types of foods are a good source of vitamins?
A1: LEGUMES are a good source of vitamins.
Q2: Are legumes a source of vitamins?
A2: Legumes are a GOOD source of vitamins.
Q3: I’ve heard that legumes are healthy, but what are they a good source of ?
A3: Legumes are a good source of VITAMINS.
Slide from Jennifer Venditti

6. Same ‘tune’, different alignment
LEGUMES are a good source of vitamins
The main rise-fall accent (= “I assert this”) shifts locations.
Slide from Jennifer Venditti

7. Same ‘tune’, different alignment
Legumes are a GOOD source of vitamins
The main rise-fall accent (= “I assert this”) shifts locations.
Slide from Jennifer Venditti

8. Same ‘tune’, different alignment
legumes are a good source of VITAMINS
The main rise-fall accent (= “I assert this”) shifts locations.
Slide from Jennifer Venditti

9. Types of Waveform Synthesis
Articulatory Synthesis:
Model movements of articulators and acoustics of vocal tract
Concatenative Synthesis:
Use databases of stored speech to assemble new utterances.
Diphone
Unit Selection
Statistical (HMM) Synthesis
Trains parameters on databases of speech
Rule-Based (Formant) Synthesis:
Start with acoustics, create rules/filters to create waveform

10. Articulatory Synthesis
Simulation of physical processes of human articulation
Wolfgang von Kempelen ( ) and others used bellows, reeds and tubes to construct mechanical speaking machines
Modern versions “simulate” electronically the effect of articulator positions, vocal tract shape, etc on air flow.

11. Concatenative approaches
Two main approaches:
1- Concatenating Phone Units
Example: concatenating samples of recorded diphones or syllables
2- Unit selection
Uses several samples for each phone unit and selects the most appropriate one when synthesizing

12. Phone Units Paragraph ( ) Sentence ( )
Word (Depends on the language. Usually more than 100,000)
Syllable
Diphone & Triphone
Phoneme (Between 10 , 100)

13. Phone Units (Cont’d)
Diphone : We model Transitions between two phonemes p1 p2 p3 p4 p5
Diphone
Phoneme

14. Phone Units (Cont’d) Farsi phonemes: 30 Farsi diphones: 30*30 = 900
Phoneme /zho/ is missing (?)
Farsi triphones: in theory
Not all of the triphones are used

15. Phone Units (Cont’d) Syllable = Onset (Consonant) + Rhyme
Syllable is a set of phonemes that exactly contains one vowel
Syllables in Farsi : CV , CVC , CVCC
We have about 4000 Syllables in Farsi
Syllables in English :V, CV , CVC ,CCVC, CCVCC, CCCVC, CCCVCC, . . .
Number of Syllables in English is too many

16. Phone Sequence To Speech (Cont’d)
to primitive
utterance
primitive
utterance
to Natural
Speech
Text to
Phone
Sequence
Speech
Text
NLP
Speech Processing

17. Concatenative Approaches
In this approaches we store units of natural speech for reconstruction of desired speech
We could select the appropriate phone unit for speech synthesis
we can store compressed parameters instead of main waveform

18. Concatenative Approaches (Cont’d)
Benefits of storing compressed parameters instead of main waveform
Less memory use
General state instead of a specific stored utterance
Generating prosody easily

19. Concatenative Approaches (Cont’d)
Phone Unit
Type of Storing
Paragraph
Sentence
Word
Syllable
Diphone
Phoneme
Main Waveform
Coded/Main Waveform
Coded Waveform

20. Concatenative Approaches (Cont’d)
Pitch Synchronous Overlap-Add-Method (PSOLA) is a famous method in phoneme transmit smoothing
Overlap-Add-Method is a standard DSP method
PSOLA is a base action for Voice Conversion.
In this method in analysis stage we select frames that are synchronous by pitch markers.

21. Diphone Architecture Example
Training:
Choose units (kinds of diphones)
Record 1 speaker saying 1 example of each diphone
Mark the boundaries of each diphones,
cut each diphone out and create a diphone database
Synthesizing an utterance,
grab relevant sequence of diphones from database
Concatenate the diphones, doing slight signal processing at boundaries
use signal processing to change the prosody (F0, energy, duration) of selected sequence of diphones

22. Unit Selection
Same idea as concatenative synthesis, but database contains bigger varieties of “phone units” from diphones to sentences
Multiple examples of phone units (under different prosodic conditions) are recorded
Selection of appropriate unit therefore becomes more complex, as there are in the database competing candidates for selection

23. Unit Selection
Unlike diphone concatenation, little or no signal processing applied to each unit
Natural data solves problems with diphones
Diphone databases are carefully designed but:
Speaker makes errors
Speaker doesn’t speak intended dialect
Require database design to be right
If it’s automatic
Labeled with what the speaker actually said
Coarticulation, schwas, flaps are natural
“There’s no data like more data”
Lots of copies of each unit mean you can choose just the right one for the context
Larger units mean you can capture wider effects

24. Unit Selection Issues Given a big database
For each segment (diphone) that we want to synthesize
Find the unit in the database that is the best to synthesize this target segment
What does “best” mean?
“Target cost”: Closest match to the target description, in terms of
Phonetic context
F0, stress, phrase position
“Join cost”: Best join with neighboring units
Matching formants + other spectral characteristics
Matching energy
Matching F0

25. Unit Selection Search

26. Joining Units
unit selection, just like diphone, need to join the units
Pitch-synchronously
For diphone synthesis, need to modify F0 and duration
For unit selection, in principle also need to modify F0 and duration of selection units
But in practice, if unit-selection database is big enough (commercial systems)
no prosodic modifications (selected targets may already be close to desired prosody)

27. Unit Selection Summary
Advantages
Quality is far superior to diphones
Natural prosody selection sounds better
Disadvantages:
Quality can be very bad in some places
HCI problem: mix of very good and very bad is quite annoying
Synthesis is computationally expensive
Needs more memory than diphone synthesis

28. Rule-Based Approach Stages
Determine the speech model and model parameters
Determine type of phone units
Determine some parameter amount for each phone unit
Substitute sequence of phone units by its equivalent parameter sequence
Put parameter sequence in speech model

29. KLATT 80 Model

30. KLATT 88 Model

31. THE KLSYN88 CASCADE PARALLEL FORMANT SYNTHESIZER
FNP FNZ FTP FTZ F1 B1
BNP BNZ BTP BTZ DF1 DB F2 B F3 B F4 B F5 B5
GLOTTAL SOUND SOURCES
NASAL
POLE ZERO PAIR
TRACHEAL POLE ZERO PAIR
FIRST FORMANT RESONATOR
SECOND FORMANT RESONATOR
THIRTH FORMANT RESONATOR
FOURTH FORMANT RESONATOR
FIFTH FORMANT RESONATOR
FILTERED IMPULSE TRAIN TL
CASCADE VOCAL TRACT MODEL LARYNGEAL SOUND SOURCES
F0 AV OO FL DI
SPECTRAL TILT LOW-PAS RESONANTOR
KL GLOTT 88 model (default) SS CP +
NASAL FORMANT RESONATOR AH
ANV
ASPIRATION NOISE GENERATOR SO
MODIFIED LF
MODEL
FIRST FORMANT RESONATOR
A1V
SECOND FORMANT RESONATOR
B2F + -
A2F
FIRST DIFFERENCE PREEMPHASIS
SECOND FORMANT RESONATOR
A2V +
THIRD FORMANT RESONATOR
B3F
A3F
THIRTH FORMANT RESONATOR AF
A3V
FRICATION NOISE GENERATOR
FOURTH FORMANT RESONATOR
B4F
A4F
FOURTH FORMANT RESONATOR
A4V
FIFTH FORMANT RESONATOR
B5F + -
A5F
TRACHEAL FORMANT RESONATOR
ATV
B6F F6
SIXTH FORMANT RESONATOR
A6F AB
PARALLEL VOCAL TRACT MODEL LYRYNGEAL
SOUND SOURCES (NORMALLY NOT USED)
BYPASS PATH
PARALLEL VOCAL TRACT MODEL FRICATION SOUND SOURCES

32. Three Voicing Source Model In KLATT 88
The old KLSYN impulsive source
The KLGLOTT88 model
The modified LF model

33. HMM-Based Synthesis Corpus-based, statistical parametric synthesis
Proposed in mid-'90s, becomes popular since mid-'00s
Large data + automatic training => Automatic voice building
Source-filter model + statistical acoustic model Flexible to change its voice characteristics
HMM as its statistical acoustic model
We focus on HMM-based speech synthesis

35. First extract parametric representations of speech including spectral and excitation parameters from a speech database
Model them by using a set of generative models (e.g., HMMs)
Training
Synthesis

36. Speech Parameter Modeling Based on HMM
Spectral parameter modeling
Excitation parameter modeling
State duration modeling

37. Spectral parameter modeling
Mel-cepstral analysis has been used for spectral estimation
A continuous density HMM is used for the vocal tract modeling in the same way as speech recognition systems.
The continuous density Markov model is a finite state machine which makes one state transition at each time unit (i.e. frame). First, a decision is made to which state to succeed (including the state itself). Then an output vector is generated according to the probability density function (pdf) for the current state

38. Cont’d…

39. F0 parameter modeling
While the observation of F0 has a continuous value in the voiced region, there exists no value for the unvoiced region. We can model this kind of observation sequence assuming that the observed F0 value occurs from one-dimensional spaces and the “unvoiced” symbol occurs from the zero-dimensional space.

41. Calculation of dynamic feature:
As was mentioned, mel-cepstral coefficient is used as spectral parameter, Their dynamic feature Δc and Δ2c are calculated as follows:
Dynamic features for F0: In unvoiced region, pt, Δpt and Δ2pt are defined as a discrete symbol.
When dynamic features at the boundary between voiced and unvoiced cannot be calculated, they are defined as a discrete symbol.

42. Effect of dynamic feature
By using dynamic features, the generated speech parameter vectors reflect not only the means of static and dynamic feature vectors but also the covariances of those
Estimation will be
smoother
Good and bad effect

43. Multi-Stream HMM structure:
The sequence of mel-cepstral coefficient vector and F0 pattern are modeled by a continuous density HMM and multi-space probability distribution HMM
Putting all this together has some advantages

44. Synthesis part
An arbitrarily given text to be synthesized is converted to a context-based label sequence.
The text is converted a context dependent label sequence by a text analyzer.
For the TTS system, the text analyzer should have the ability to extract contextual information. However, no text analyzer has the ability to extract accentual phrase and to decide accent type of accentual phrase.

45. Some contextual factors
When we have HMM for each phoneme
{preceding, current, succeeding} phoneme
position of breath group in sentence
{preceding, current, succeeding} part-of-speech
position of current accentual phrase in current breath group
position of current phoneme in current accentual phrase

47. According to the obtained state durations, a sequence of mel-cepstral coefficients and F0 values including voiced/unvoiced decisions is generated from the sentence HMM by using the speech parameter generation algorithm
Finally, speech is synthesized directly from the generated mel-cepstral coefficients and F0 values by the MLSA filter

48. Spectral representation & corresponding filter
cepstrum: LMA filter
generalized cepstrum: GLSA filter
mel-cepstrum: MLSA (Mel Log Spectrum Approximation) filter
mel-generalized cepstrum: MGLSA filter
LSP: LSP filter
PARCOR: all-pole lattice filter
LPC: all-pole filter

49. Advantages
Most of the advantages of statistical parametric synthesis against unit-selection synthesis are related to its flexibility due to the statistical modeling process.
Transforming voice characteristics, speaking styles, and emotions.
Also Combination of unit selection and voice conversion (VC) techniques can alleviate this problem, high-quality voice-conversion is still problematic.

50. Adaptation (mimicking voices):Techniques of adaptation were originally developed in speech recognition to adjust general acoustic model , These techniques have also been applied to HMM-based speech synthesis to obtain speaker-specific synthesis systems with a small amount of speech data
Interpolation (mixing voices): Interpolate parameters among representative HMM sets
- Can obtain new voices even no adaptation data is available
- Gradually change speakers. & speaking styles

51. Some rule-based applications
KLATT C Implementation:
A web page which generates speech waveform online given KLATT parameters:
tutorials/synthesis/Klatt.html
Formant Synthesis Demo
using Fant’s Formant Model
wavesurfer/formant/

52. TTS Online Demos AT&T: Festival Cepstral IBM
Festival
Cepstral
IBM

53. Festival Open source speech synthesis system
Designed for development and runtime use
Use in many commercial and academic systems
Distributed with RedHat 9.x, etc
Hundreds of thousands of users
Multilingual
No built-in language
Designed to allow addition of new languages
Additional tools for rapid voice development
Statistical learning tools
Scripts for building models
1/5/07
Text from Richard Sproat

54. Festival as software http://festvox.org/festival/
General system for multi-lingual TTS
C/C++ code with Scheme scripting language
General replaceable modules:
Lexicons, LTS, duration, intonation, phrasing, POS tagging, tokenizing, diphone/unit selection, signal processing
General tools
Intonation analysis (f0, Tilt), signal processing, CART building, N-gram, SCFG, WFST
1/5/07
Text from Richard Sproat

55. Festival as software http://festvox.org/festival/ No fixed theories
New languages without new C++ code
Multiplatform (Unix/Windows)
Full sources in distribution
Free software
1/5/07
Text from Richard Sproat

56. CMU FestVox project Festival is an engine, how do you make voices?
Festvox: building synthetic voices:
Tools, scripts, documentation
Discussion and examples for building voices
Example voice databases
Step by step walkthroughs of processes
Support for English and other languages
Support for different waveform synthesis methods
Diphone
Unit selection
Limited domain
1/5/07
Text from Richard Sproat

57. Future Trends
Speaker adaptation
Language adaptation