Emotional Speech Julia Hirschberg CS 6998 2/16/2019

Today Defining emotional speech Emotional categories Eliciting judgments Producing emotional speech Detecting emotional speech A Subclass: Deceptive speech 2/16/2019

Cowie ‘00 Is there a good theoretical or practical definition of emotional speech? “Full-blown” emotion vs. emotional state Cause and effect descriptions Primary and secondary (second order) Everyday descriptions Representations Biological 2/16/2019

Dimensions in continuous space, e.g. Valence: positive or negative Activation level: how disposed to take action Structural models: different ways of appraising situation that evokes emotion e.g. positive or negative? Does situation help agent to achieve his/her goals? Timing as a key variable sadness vs. grief vs. depression vs. gloominess 2/16/2019

How are emotions expressed? Display rules? In speech? Mixing Simulation 2/16/2019

Schroeder ‘01: Emotion in Synthesis How is a given emotion expressed in speech? What are the properties of the emotion to be expressed? How are they related to those of other emotions? What kind of synthesizer works best? Formant Diphone Unit selection 2/16/2019

Prosody rules: what to modify? How do we evaluate the results? Forced choice Free response Recognition rate Perceived naturalness 2/16/2019

Ten Bosch ‘00: Emotion Recognition How hard is the problem? Is ‘standard’ ASR technology well-suited to it? Acoustic and language models target short local events Feature extraction normlizes/excludes e.g. pitch, rate, amplitude -- why? Interaction: emotional speech and ASR performance Synthesis needs one good example but... 2/16/2019

Ang et al Challenges: Use output from ASR system Use automatic prosodic features Find good speaker normalization Combine with lexical features Pioneered approach of “direct modeling” – no use of intermediate phonological units Applications: detecting frustration, disappointment/tiredness, amusement/surprise Results: prediction comparable to human accuracy 70-75% 2/16/2019

Method: Prosodic Models Extract pitch from signal Speech recognizer outputs word and phone alignments (duration features) Utterance-level features extracted (e.g., max speaker normalized pitch in the longest phone-normalized vowel, etc) Decision trees created to provide posterior probabilities of emotion classes given features Feature selection from development test set Separate test set used for evaluation 2/16/2019

Prosodic Features Duration features Phone / Vowel / Syllable Durations Normalized by Phone/Vowel Means, Speaker Speaking rate features (vowels/time) Pause features Speech to pause ratio, number of long pauses Maximum pause length Energy features (RMS energy) Pitch features Used pitch stylization algorithm (Sonmez et al.) LTM model of F0 to estimate speaker range Pitch ranges, slopes, locations of interest Spectral tilt features Other (non-prosodic) features Position of utterance in dialog Repeat or correction 2/16/2019

Emotion in Deception Motivation: why might such cues exist? Deception evokes emotion in deceivers (e.g. Ekman ‘85-92) Fear of discovery: higher pitch, faster, louder, pauses disfluencies, indirect speech Elation at successful deceiving: higher pitch, faster, louder, greater elaboration 2/16/2019

Acoustic/Prosodic/Lexical Cues Are deceivers less forthcoming? Shorter speech with fewer details Are lies less compelling than truths? Less plausible, logical, more discrepancies Less verbal and vocal ‘involvement’ Less verbal ‘immediacy’: more passives, negations, indirect speech More uncertainty (subjective) More repetitions Are liars less positive, pleasant? 2/16/2019

More negative statements, complaints Are liars more tense? Nervous overall Vocal tension High pitch Do lies contain fewer ‘imperfections’? Fewer self-repairs Fewer admissions of forgetfulness Fewer scene descriptions, details More mention of peripheral events or relationships 2/16/2019

Current State-of-the-Art No single cue to deceptive speech: most studied are visual Other acoustic/prosodic features proposed, but evidence mixed so far Loudness/intensity Speaking rate Response latency Disfluencies No attested method to detect deception automatically using acoustic/prosodic/lexical cues All current findings are descriptive, suggestive All proposed methods require human intervention 2/16/2019

Our Approach Elicit deceptive and non-deceptive corpus Motivation: Identity-relevant (self-image) and instrumental (monetary) incentives “Real” deception vs. acted Good recording conditions Tasks/interview paradigm Transcription/annotation Acoustic/prosodic/lexical analysis to identify features of interest, test validity of paradigm Automatic feature extraction and analysis to train models of deceptive and non-deceptive speech 2/16/2019

Corpus Collection Subjects asked to perform tasks for comparison with target profile of 25 top entrepreneurs Performance manipulated to produce performance same as/differing from target Monetary incentive to convince an interviewer they matched target Recorded interview/interrogation Biographical information (t/f) “Big lie” on task performance “Local lie”: Pedal indicators of t/f for each answer 2/16/2019

To date: 15 subjects, totaling ~3h of subject speech Collection To date: 15 subjects, totaling ~3h of subject speech Planned: 7-8h hours of subject speech 2/16/2019

Results of Prosodic/Acoustic Analysis On Arizona Mock Theft data subset: 32 interviews/72m, required segmentation, recording issues (50/160m more being segmented) Significant pitch feature differences between deceptive and non-deceptive speech, but... Highly motivated speakers lower pitch when lying Low motivation speakers raise pitch when lying Males lower pitch when lying Females raise pitch when lying 2/16/2019

Preliminary analyses of 8 speakers for ‘local’ t/f On Columbia corpus: Preliminary analyses of 8 speakers for ‘local’ t/f Significant differences in pitch range for six subjects, but differ from Mock Theft wrt gender Lexical findings: Preliminary analyses on Columbia data using LIWC show negative words more prevalent in deceptive speech 2/16/2019