1. The Creation of Emotional Effects for An Arabic Speech Synthesis System
Prepared by:
Waleed Mohamed Azmy
Under Supervision:
Prof. Dr. Mahmoud Ismail Shoman
Dr. Sherif Mahdy Abdou

2. Agenda Motivations & Applications Emotional Synthesis Approaches
Unit Selection & Blending Data Approach
Arabic Speech Database & Challenges
Festival – Speech Synthesis framework
Arabic Voice Building
Proposed Utterance Structure
Proposed Target Cost Function
System Evaluation & Results
Conclusion & Future Works

3. Motivations & Applications
Emotions are inseparable components of the natural human speech.
Because of that, the level of human speech can only be achieved with the ability to synthesize emotions.
As the tremendous increasing in the demand of speech synthesis systems, Emotional synthesis become one of the major tends in this field.

4. Motivations & Applications
Emotional speech synthesis can be applied to different applications like:
Spoken Dialogue Systems
Customer-care centers
Task planning
Tutorial systems
Automated agents
Future trends are towards approaching Artificial Intelligence

5. Emotional Synthesis Approaches
Attempts to add emotion effects to synthesised speech have existed for more than a decade.
Most emotional synthesis approaches inherent properties of the various normal synthesis techniques used.
Different emotional synthesis techniques provide control over acoustic parameters to very different degrees.
Important emotional related acoustic parameters like: Pitch, Duration, speaking rate …etc

6. Emotional Synthesis Approaches
Emotional Formant Synthesis
Also known as rule-based synthesis
No human speech recordings are involved at run time.
The resulting speech sounds relatively unnatural and “robot-like”.
Emotional Diphone Concatenation
Use Diphone concatenation on Emotional recordings
A majority reports shows a degree of success in conveying some emotions.
May harm voice quality

7. Emotional Synthesis Approaches
HMM-Based Parametric synthesis
Use HMM models to change speaking style and emotional expression of the synthetic speech.
Train models on speech produced with different speaking styles.
Requires several hundreds of sentences spoken in a different speaking style.
Emotional-Based Unit-Selection
Use variable length units
Units that are similar to the given target are likely to be selected.
It is the used approach in our proposed system.

8. Emotional Unit Selection
Units are selected from large databases of natural speech.
The quality of unit selection synthesis depends heavily on the quality of the speaker and on the coverage of the database.
The quality of the synthesis relies on the fact that little or no signal processing is done on the selected units.
Two major cost functions are used
Joint Cost function
Target Cost function

9. Blending Data Approach
Two main techniques of building emotional voices for unit selection
Tiering techniques
Limited domain synthesis.
Requires a large number of databases for different emotions.
Blending techniques
General purpose application.
All databases are located in one data pool used for selection.
… used technique in our system.
In blending approach, The system will choose the units from only one Database.
Requires careful selection criterion to match target speaking style.

10. Arabic Speech Database
In our system we used RDI TTS Saudi speaker database.
This database consists of 10 hours of recording with neutral emotion and one hour of recordings for four different emotions that are sadness, happiness, surprise and questioning.
The EGG signal is also recorded to support pitch marking during the synthesis process.
The database is for male speaker sampled with 16 kHZ sampling rate.
HMMs based Viterbi alignment procedure is used to produce the phone level segmentation boundaries

11. Festival – Speech Synthesis framework
The Festival TTS system was developed at the University of Edinburgh by Alan Black and Paul Taylor.
Festival is primarily a research toolkit for speech synthesis (TTS).
It provides a state-of-the-art unit selection synthesis module called “Multisyn”.
Festival uses a data structure called an utterance structure that consists of items and relations.
An utterance represents some chunk of text that is to be rendered as speech

12. Festival – Speech Synthesis framework
Example of utterance structure:

13. Arabic Voice Building
Voice building phase is one of the major steps in building a unit selection synthesizer.
The voice building toolkit that comes with festival has been used for building our Arabic Emotional voice. The following steps were used:
Generate power factors and wave file normalization
Using EGGs for accurate pitch marking extraction
Generating LPC and residuals from wave files
Generate MFCCs, pitch values and spectral coefficients

14. Arabic Voice Building(Pitch marking)
In unit selection; Accurate estimation of pitchmarks is necessary for pitch modification to assure optimal quality of synthetic speech.
We used EGG signal to extract pitchmarks.
Pitchmarking enhancements has been carried out by matrix optimization process.
The low pass filter and high pass filter cut-off frequencies have been chosen for the optimization process.

15. Arabic Voice Building(Pitch marking)
Example of the default parameters of the pitchmarking application versus the optimized ones.
Default Pitchmarking
Optimized Pitchmarking

16. Proposed Utterance Structure
The HMM-based alignment transcription is converted to the utterance structure.
ASMO-449 is used to transliterate the Arabic text to English characters in the utterance files.
The utterance structures of both the utterances in the training databases and the target utterances is changed to carry emotional information.
A new feature called emotion has been added to the word item type in the utterance structure.

17. Proposed Utterance Structure
The proposed system is designed to work with three emotional state (normal, sad and question).
So, The emotion feature takes one of three emotional values.
Normal  Normal state
Sad  Sad emotion state
Question  Question emotion sate

18. Proposed Target Cost Function
The target cost is a weighted sum of functions that check if features in the target utterance match those features in the candidate utterance.
The standard target cost in festival does not contain any computation differences for emotional state of the utterance.
A new emotional target cost is proposed

19. Proposed Emotional Target Cost Function
The target cost:
The Emotional target cost:

20. Proposed Target Cost Function
The algorithm in general favors units that are similar in the emotional sate or classified to be emotional in a two stages of penalties.
The tuning weighting factor of the emotional target cost is optimized by try- and-error to find appropriate value.

21. System Evaluation
Six sentences were synthesized for each emotional state.
The Sentences were from the news websites, usual conversations and the holy Qur’an.
Two major set of evaluation used
Deterministic Evaluation
Perceptual Evaluation
In deterministic evaluation tests the emotional classifier system Emovoice is used across the output utterance.
Emovoice is trained on the Arabic data using SVM model and its complete feature set.

22. System Evaluation In perceptual tests, 15 participant have involved
Two types of listening tests were performed in order to evaluate the system perceptually.
Test intelligibility
The listener was asked to type in what they heard
Word Error Rate (WER) is computed

23. System Evaluation Test Naturalness & Emotiveness
The actual experiment took place on a computer where subjects had to listen to the synthesized sentences using headphones.
The listeners are asked to rate the quality of the output voice in terms of (Naturalness & Emotiveness)
Participants were asked to give ratings between 1 and 4 for poor, acceptable, good and excellent respectively.

24. System Evaluation - Results
The confusion matrix of the classifier output emotion and the target emotional state of the utterance is
Classified As
Normal
Sad
Question 5 1 6

25. System Evaluation - Results
The results of the WER is shown in figure.
It shows a maximum average of 8% in sad emotion
WER

26. System Evaluation - Results
The descriptive statistics of the naturalness and emotiveness ratings is summarized..
Naturalness
Emotiveness
Rating Mean
Standard Deviation
Normal
2.72
1.01
Sad
2.71
1.02
Question
2.56
0.96
All
2.67
1.06
Rating Mean
Standard Deviation
Normal
3.28
0.71
Sad
2.76
1.09
Question
3.08
0.81
All
3.04
0.95

27. System Evaluation - Results
The overall mean of naturalness ratings is 2.7 which approach a good quality naturalness.
The average ratings of overall emotiveness are 3.1 which indicate good emotive state of the synthesized speech
The naturalness and emotiveness ratings for question emotion sentences has lower mean value and high variance which means that they were not –to some extent- recognized as natural human speech. However it shows a good emotiveness scores.

28. Conclusions
The main goal of this research was to develop an Emotional Arabic TTS voice.
This research focused on three important emotional sates; normal, sad and questions.
According to the different tests performed on the system, it shows promising results. At most the participants feel acceptable natural voice with clear good emotive state.

29. Future works
It is recommended to increase the duration of acted or real emotional utterances in the RDI Arabic speech database.
However the work done for accurate pitch- marking, some further enhancements are needed especially for question speech utterances.
Optimizing the pitch modification module in festival for better concatenation with different emotions.
Use emotional speech conversion based on signal processing and feature modelling technologies. The initial key features are commonly known to be pitch and duration

30. Questions ?