2. 1 Computation Approaches to Emotional Speech Julia Hirschberg julia@cs.columbia.edu

3. 2 Why Study Emotional Speech? Recognition Anger/frustration in call centers Confidence/uncertainty in online tutoring systems “Hot spots” in meetings Generation TTS for –Computer games –IVR systems Other applications: Speaker State –Deception, Charisma, Sleepiness, Interest… –The Love Detector (available for Skype )…The Love Detector

4. 3 Assessing Health-Related Conditions Assessing intoxication levels (Levit et al ‘01) Distinguishing between active and passive coping responses in patients with breast cancer (Zei Pollermann ’02) Assessing schizophrenia (Bitouk et al ‘09) Classifying degree of autistic behavior (Columbia) Suicide notes

5. 4 Hard Questions in Emotion Recognition How do we know what emotional speech is? –Acted speech vs. natural (hand labeled) corpora What can we classify? Distinguish among multiple ‘classic’ emotions Distinguish –Valence: is it positive or negative? –Activation: how strongly is it felt? (sad/despair) What features best predict emotions? What techniques best to use in classification?

6. 5 happy sad angry confident frustrated friendly interested anxious bored encouraging Acted Speech: LDC Emotional Speech CorpusLDC Emotional Speech Corpus

7. 6 Is Natural Emotion Different? (thanks to Liz Shriberg) Neutral –July 30 –Yes Disappointed/tired –No Amused/surprised –No Annoyed –Yes –Late morning Frustrated –Yes –No –No, I am … –…no Manila...

8. 7 Major Problems for Classification: Different Valence/Different Activation

9. 8 But…. Different Valence/ Same Activation

10. 9 Good Features Can be Hard to Find Useful features: –Automatically extracted pitch, intensity, rate, VQ –Hand-labeled, automatically stylized pitch contours –Context –Lexical information: Dictionary of Affect –But….individual and cultural differences Algorithms for classification: –Machine learning (Decision trees, Support Vector Machines, Rule induction algorithms, HMMs,…)

11. 10 Results: Different Emotions, Different Success Rates EmotionBaselineAccuracy angry69.32%77.27% confident75.00% happy57.39%80.11% interested69.89%74.43% encouraging52.27%72.73% sad61.93%80.11% anxious55.68%71.59% bored66.48%78.98% friendly59.09%73.86% frustrated59.09%73.86%

12. 11 Open Questions New features and algorithms New types of emotion/speaker state to identify New ways of finding/collecting useful data New applications of more-or-less successful emotion classification Interspeech Paralinguistic Challenges

13. 12 This Class Goals: –Learn what we know about: readings and discussion participation –Learn how to analyze speech, how to design a speech experiment, how to classify speaker states –Try to contribute something new: term project –Practice doing research Syllabus: –http://www.cs.columbia.edu/~julia/courses/CS6998/syll abus11.htm

14. 13 Readings and Discussion Weekly readings –Everyone prepares/hands in 3 discussion questions on each assigned paper or website If you read an optional paper, submit questions on that as well if you want ‘credit’ –Everyone participates in class discussion –Each week one person leads discussion on one paper –Submit pdf in courseworks shared files

15. 14 Term Project Everyone prepares a term project on a topic of their choice –You may work alone or in teams of 2 Deliverables –Proposal –Interim progress report –Final report –Short presentation/demo

16. 15 Possible Topics Collect audio from children of different ages winning and losing a game and see if adults can distinguish those who win (happy speech) from those who lose (sad speech). Create hybrid speech stimuli from tokens uttered with different emotions (mixing pitch, loudness, duration, speaking rate,...) and see which features of emotional speech are most reliably associated with emotions. Detect different emotions from Cantonese and Mandarin speakers and compare performance of an automatic program to performance of human judges. Train Machine Learning algorithms on emotional speech corpora and see if you can improve over other approaches on the same corpora Develop an email reader that detects emotion from text and uses the appropriate emotional TTS system to read it to the use

17. 16 Important Details Read the academic integrity paragraph in the syllabus and understand it. Do all the readings when they are due, turn in all discussion questions by noon on the day of class, come to every class

18. 17 Questions?