Acoustic-Prosodic and Lexical Entrainment in Deceptive Dialogue

Slides:



Advertisements
Similar presentations
1 Using the HTK speech recogniser to analyse prosody in a corpus of German spoken learners English Toshifumi Oba, Eric Atwell University of Leeds, School.
Advertisements

Identifying Deceptive Speech Across Cultures (FA ) PI: Julia Hirschberg (Columbia University) Co-PI: Andrew Rosenberg (CUNY) Co-PI: Michelle.
The Role of F0 in the Perceived Accentedness of L2 Speech Mary Grantham O’Brien Stephen Winters GLAC-15, Banff, Alberta May 1, 2009.
Human Speech Recognition Julia Hirschberg CS4706 (thanks to John-Paul Hosum for some slides)
“Effect of Genre, Speaker, and Word Class on the Realization of Given and New Information” Julia Agustín Gravano & Julia Hirschberg {agus,
SPEECH RECOGNITION 2 DAY 15 – SEPT 30, 2013 Brain & Language LING NSCI Harry Howard Tulane University.
Combining Prosodic and Text Features for Segmentation of Mandarin Broadcast News Gina-Anne Levow University of Chicago SIGHAN July 25, 2004.
Comparing American and Palestinian Perceptions of Charisma Using Acoustic-Prosodic and Lexical Analysis Fadi Biadsy, Julia Hirschberg, Andrew Rosenberg,
Presented by Ravi Kiran. Julia Hirschberg Stefan Benus Jason M. Brenier Frank Enos Sarah Friedman Sarah Gilman Cynthia Girand Martin Graciarena Andreas.
Prosodic Cues to Discourse Segment Boundaries in Human-Computer Dialogue SIGDial 2004 Gina-Anne Levow April 30, 2004.
Spoken Language Processing Lab Who we are: Julia Hirschberg, Stefan Benus, Fadi Biadsy, Frank Enos, Agus Gravano, Jackson Liscombe, Sameer Maskey, Andrew.
Ability to attract and retain followers by virtue of personal characteristics - not traditional or political office (Weber ‘47) What makes an individual.
Modeling Other Speaker State COMS 4995/6998 Julia Hirschberg Thanks to William Wang.
Extracting Social Meaning Identifying Interactional Style in Spoken Conversation Jurafsky et al ‘09 Presented by Laura Willson.
High Frequency Word Entrainment in Spoken Dialogue ACL, June Columbus, OH Department of Computer and Information Science University of Pennsylvania.
Context and Prosody in the Interpretation of Cue Phrases in Dialogue Julia Hirschberg Columbia University and KTH 11/22/07 Spoken Dialog with Humans and.
Turn-taking in Mandarin Dialogue: Interactions of Tone and Intonation Gina-Anne Levow University of Chicago October 14, 2005.
Classification of Discourse Functions of Affirmative Words in Spoken Dialogue Julia Agustín Gravano, Stefan Benus, Julia Hirschberg Shira Mitchell, Ilia.
9/5/20051 Acoustic/Prosodic and Lexical Correlates of Charismatic Speech Andrew Rosenberg & Julia Hirschberg Columbia University Interspeech Lisbon.
10/10/20051 Acoustic/Prosodic and Lexical Correlates of Charismatic Speech Andrew Rosenberg & Julia Hirschberg Columbia University 10/10/05 - IBM.
Varying Input Segmentation for Story Boundary Detection Julia Hirschberg GALE PI Meeting March 23, 2007.
Agustín Gravano 1,2 Julia Hirschberg 1 (1)Columbia University, New York, USA (2) Universidad de Buenos Aires, Argentina Turn-Yielding Cues in Task-Oriented.
Schizophrenia and Depression – Evidence in Speech Prosody Student: Yonatan Vaizman Advisor: Prof. Daphna Weinshall Joint work with Roie Kliper and Dr.
Advanced Supplementary Level. You learn how to: 4 write grammatical English 4 speak in English (for communication) 4 read English for general understanding.
Is phonetic variation represented in memory for pitch accents ? Amelia E. Kimball Jennifer Cole Gary Dell Stefanie Shattuck-Hufnagel ETAP 3 May 28, 2015.
Use of phonetic specificity during the acquisition of new words: Differences between consonants and vowels. Thiery Nazzi (2004) By: Dominique, Jennifer,
On Speaker-Specific Prosodic Models for Automatic Dialog Act Segmentation of Multi-Party Meetings Jáchym Kolář 1,2 Elizabeth Shriberg 1,3 Yang Liu 1,4.
Yun-Nung (Vivian) Chen, Yu Huang, Sheng-Yi Kong, Lin-Shan Lee National Taiwan University, Taiwan.
Evaluating prosody prediction in synthesis with respect to Modern Greek prenuclear accents Elisabeth Chorianopoulou MSc in Speech and Language Processing.
Epenthetic vowels in Japanese: a perceptual illusion? Emmanual Dupoux, et al (1999) By Carl O’Toole.
(2) Using age-appropriate activities, students develop the ability to perform the tasks of the novice language learner. The novice language learner, when.
Na1c0014 李羿霈.  An acoustic perspective of English vowel production and perception by Taiwanese EFL learners, as compared with native speakers of English.
Recognizing Discourse Structure: Speech Discourse & Dialogue CMSC October 11, 2006.
National Taiwan University, Taiwan
Predicting Voice Elicited Emotions
1/17/20161 Emotion in Meetings: Business and Personal Julia Hirschberg CS 4995/6998.
What do English language learners need to know and be able to do when they leave my class? August 2007 Training Guide Session II 1.
Acoustic Cues to Emotional Speech Julia Hirschberg (joint work with Jennifer Venditti and Jackson Liscombe) Columbia University 26 June 2003.
1 Identifying Deceptive Speech within and across Cultures Sarah Ita Levitan, Guozhen An, Michelle Levine, Andrew Rosenberg, Julia Hirschberg Computer.
Prosodic Cues to Disengagement and Uncertainty in Physics Tutorial Dialogues Diane Litman, Heather Friedberg, Kate Forbes-Riley University of Pittsburgh.
On the role of context and prosody in the interpretation of ‘okay’ Julia Agustín Gravano, Stefan Benus, Julia Hirschberg Héctor Chávez, and Lauren Wilcox.
1 Deceptive Speech: Acoustic, Prosodic Lexical, Gender, Ethnicity, and Personality Factors Julia Hirschberg Computer Science Columbia University 14 June.
Investigating Pitch Accent Recognition in Non-native Speech
Chapter 3: The variation problem 2: Intra-speaker variation
Hyperscanning Deception: EEG Analysis of the Lying Game
Intonation and Computation: Deception
WTC, Native-Speakerism, and TOEIC Scores
Why Study Spoken Language?
Studying Intonation Julia Hirschberg CS /21/2018.
Studying Intonation Julia Hirschberg CS /21/2018.
Linguistic Relativity: Evidence from Native Korean and English Speakers and Factors Affecting Its Extent.
Comparing American and Palestinian Perceptions of Charisma Using Acoustic-Prosodic and Lexical Analysis Fadi Biadsy, Julia Hirschberg, Andrew Rosenberg,
Why Study Spoken Language?
Turn-taking and Disfluencies
Recognizing Structure: Sentence, Speaker, andTopic Segmentation
Fadi Biadsy. , Andrew Rosenberg. , Rolf Carlson†, Julia Hirschberg
High Frequency Word Entrainment in Spoken Dialogue
Agustín Gravano & Julia Hirschberg {agus,
Advanced NLP: Speech Research and Technologies
Discourse Structure in Generation
Agustín Gravano1,2 Julia Hirschberg1
ANJANA RAJ English Optional
Emotional Speech Julia Hirschberg CS /16/2019.
The Development of the E 8 Listening Test
TOEFL.
Tools for Speech Analysis
Globalization & World Languages
Low Level Cues to Emotion
Guest Lecture: Advanced Topics in Spoken Language Processing
Automatic Prosodic Event Detection
Presentation transcript:

Acoustic-Prosodic and Lexical Entrainment in Deceptive Dialogue Sarah Ita Levitan With: Jessica Xiang, Julia Hirschberg Advanced Topics in Spoken Language Processing March 15, 2019

Research Questions Do interlocutors entrain in acoustic-prosodic and lexical dimensions in deceptive dialogues? Is entrainment related to deception outcomes? Is entrainment correlated with ability to deceive or detect deception? Is there a difference in entrainment behavior between truthful and deceptive speech?

Columbia X-Cultural Deception (CXD) Corpus Corpus of within-subject deceptive and non-deceptive speech Fake resume paradigm - interview format using 24-item biographical questionnaire Native speakers of SAE and MC, all speaking in English 170 dialogues between 340 subjects, >122 hours of speech Global and local deception annotation Global labels of perception of deception

Features 8 acoustic-prosodic features 4 lexical features Intensity {mean, max} Pitch {mean, max} Speaking rate Jitter, shimmer, NHR 4 lexical features 100, 25 top frequency words Hedge words, phrases Cue phrases

Entrainment measures Local Global Proximity   Local Global Proximity Local partner_diff < other_diff Global partner_diff < other diff Convergence Correlation between partner_diff and time partner_diff in last 5 min < partner_diff in first 5 min Synchrony Correlation between partner IPU features NA

Local Entrainment Feature Proximity Convergence Synchrony t p r   t p r Max Pitch -3.12 ** 3 NS 0.02 *** Mean Pitch 4.87 -0.006 0.03 Max Intensity 12.82 0.15 Mean Intensity 10.67 0.04 0.16 Speaking Rate 6.04 -0.01 0.08 Jitter 3.95 * 0.05 Shimmer 2.48 0.0005 NHR 2.75 0.012

Global Entrainment Feature Proximity Convergence t p High Freq. 100   t p High Freq. 100 0.33 NS 1.99 * High Freq. 25 2.56 2.05 Hedge 2.82 ** 1.29 Cue 0.18 1.18 Max Pitch 2.10 -0.56 Mean Pitch 0.89 0.14 Max Intensity 3.94 *** 0.02 Mean Intensity 4.26 -0.49 Speaking Rate 3.98 1.04 Jitter 3.20 0.37 Shimmer 3.44 1.58 NHR 2.31 0.92

Deception Analysis Is there a difference in entrainment behavior between truthful and deceptive speech? ✔ Greater local proximity in max intensity in deceptive speech than truthful speech (t(7244)=3.08; p=0.02)) ✔ Greater local proximity in jitter in truthful speech than deceptive speech (t(7226)=2.66; p=0.008))

Deception Analysis Is there a difference in entrainment behavior between speech that is trusted or mistrusted? ✔ Greater local proximity in mean intensity in speech judged to be deceptive than speech judged to be truthful (t(7222)=2.45; p=0.014)

Deception Analysis Is there a difference in entrainment behavior between successful and unsuccessful lies? ✘ No significant difference for any feature. It seems that interviewers were not able to perceive entrainment differences in truthful and deceptive speech.

Deception Analysis Is entrainment related to ability to deceive or to judge deception? ✔ Global proximity in high frequency (25) words is correlated with interviewer ability to judge deception (r=0.13; p=0.015) ✘ No significant correlation between entrainment and ability to deceive.

Conclusions Evidence of entrainment in deceptive speech in acoustic-prosodic and lexical dimensions, at both global and local levels Identified differences in entrainment behavior in truthful vs. deceptive speech, and trusted vs. mistrusted speech In future work, we will explore leveraging entrainment features for deception and trust classification tasks