1. A study on Prediction on Listener Emotion in Speech for Medical Doctor Interface M.Kurematsu Faculty of Software and Information Science Iwate Prefectural University

2. Virtual Medical Doctor System What is Virtual Medical Doctor System? –Diagnoses people like a human doctor –Interact with people like a person What should the speech interface module? –Speech Recognition Understands what people say Estimates Emotion in Speech –Speech Synthesize Tells diagnosis, Asks people Expresses Emotion in Speech Research Target

3. How to Estimate Emotion in Speech 1 Conventional Approach from Exists Works Step.1 Collect Human Speeches Human speech has Sound data and Emotion putted by Speaker Step.2 Feature Selection Step.2-1 Get Speech Features Step.2-2 Calculate Statistics Values Step.3 Make Classifiers of Speech Features Extracts Relation between Emotion and Speech Features Learning Phase Step.1 Record Human Speech Step.2 Feature Selection Get Speech Features & Calculate Statistics Values Step.3 Estimate Emotion Using Classifiers Estimate Phase

4. How to Estimate Emotion in Speech 2 Our Approach We modify the Learning Phase in Conventional Approach 1. Collect Human SpeechesPoints+ Use Emotion putted by Listeners + Use Synthetic Speech as Human Speeches 2-1. Feature Selection / Get Speech Features Points+ Focus on Features in each Syllable 2-2. Feature Selection / Calculate Statistics Value Points+ Calculate Quartile & Interquartile range + Calculate the coefficient of the regression formula 3. Make Classifiers Points+ Make a set of Classifiers make each classifier for each emotion

5. Evaluate Our Approach Based on Experiments Points of Modification in Our ApproachEvaluate Step.1+ Use Emotion putted by ListenersWeak Depends on a Listener Step.1+ Use Synthetic Speech as Human SpeechesWeak Useful for Some Emotions Step.2+ Focus on Features in each SyllableWeak Useful for Some Emotions Step.2+ Calculate Quartile & Interquartile rangeMaybe Good Step.2+ Calculate the coefficient of the regression formulaNot Good Step.3+ Make a set of ClassifiersGood We should modify this module more and more

6. Future Works about Estimation For Collect Speech –Subdivide Emotion by Expression Patterns –Collect Speeches more (Radio, TV, Movies etc.) For Feature Selection –Focus on Other Features E.g. Self-Correlation, LPC Spectrum etc. –Focus on Other Statistics Values E.g. correlation between some speech features For Make Classifiers –Using Other Machine Learning Methods E.g. Bagging

7. How does system Express Emotion in Speech? Adjust Speech features to Emotion Based on Relations between Emotion & Features –Speech Features= Pitch, Volume, Speed etc –Relation shows How does a system change speech features to express an emotion. How do we make relations? Developer defines based on his experience Extract from Speech and Emotion estimated by People –People hear speeches and estimate emotionSomet09(6) Express Emotion in Speech Our Approach

8. Extract Relations between Emotion and Speech features 1.Synthesize some speeches whose features are difference each other To synthesize speeches, we use SMARTALK powered by OKI Co. We use difference parameter set each synthesized speeches Parameter={ Volume, Speed, Pitch, Intonation } 2.People estimate emotion in synthetic speeches and answer emotion 14 men and 10 women answered 3.Defined Parameters as Relation We select a parameter set. Most people answered same emotion in a speech which synthesized with this parameter set We select 3 parameter sets for each emotion. Synthesize Speech to Express Emotion –We give a phrase and emotion to the module. –The module selects relation (a parameter set) and sets them. –The module synthesizes speechSomet09(7) How to Make our Speech Module

9. Snap Shot of Our System Text Box + Input a phrase Synthesize Button + Synthesize Speech with Emotion written on a Button + [SPEAK] means Not to express Emotion Development Environment +OS Windows XP sp3 +Language Visual C++ 6.0 +Library Smartalk.OCX or SAPI

10. Future Works about Synthesize Modify Relation (Parameter Set) –People evaluated this module We demonstrated this module in a local museum and asked the following question “Is synthesized speech like a human speech?” Answer: Yes=50,Moderately Yes=147, Moderately No= 133, No=27 –We need to modify Relation to synthesize speech like a human Change other parameters Give variety to parameters Add Stress and Pause Etc.

11. Appendix I showed a next slide on the Workshop –I showed the content of that slide in preceding slides.

12. Speech DATA Analysis Estimate Features Making Classifier Human Speech intentional mean ・ Max A Classifier for all emotions Our Approach Synthesized voice +SD + Kurtosis + Skewness Classifier for each emotion Classifier StepWell-Known Using to estimate emotion in speech Pitch & Power measure +Difference / Ratio Estimate Emotion in Speech