High Frequency Word Entrainment in Spoken Dialogue ACL, June Columbus, OH Department of Computer and Information Science University of Pennsylvania - Philadelphia, PA Department of Computer Science Columbia University - New York, NY Julia Ani Nenkova - Agustín Gravano - Julia Hirschberg

Agustín Gravano - ACL - June Entrainment In conversation, people adapt the way they speak to match their partners’. Entrainment, accommodation, adaptation, alignment, convergence.

Agustín Gravano - ACL - June Previous Work Existence of entrainment In conversation, speakers: Negotiate common ways of describing things. S.E. Brennan, 1996 Alter their intensity to match their partners’. R. Coulston et al., 2002 A. Ward & D. Litman, 2007 Reuse syntactic constructions. D. Reitter et al., 2006

Agustín Gravano - ACL - June Previous Work Role of entrainment Entrainment at different levels (lex, syn, sem): Is key for both production and understanding, and facilitates interaction. M.J. Pickering & S. Garrod, 2004 D. Goleman, 2006 Is a good predictor of task success (MapTask). D. Reitter & J. Moore, 2007

Agustín Gravano - ACL - June This Work Novel measures of entrainment based on usage of high-frequency words (HFW). Entrainment and… Perceived naturalness Task success Dialogue coordination Implications in the development of Spoken Dialogue Systems.

Agustín Gravano - ACL - June High-Frequency Words Most common words in a corpus, or in a conversation. Typically, function words and cue words. Entrainment of HFW Domain-independent

Agustín Gravano - ACL - June Entrainment & Naturalness Will a conversation be perceived as more natural if HFW entrainment occurs? Switchboard corpus 2430 spontaneous telephone conversations in American English Speakers asked to discuss a pre-assigned topic Annotated for degree of perceived naturalness, from “1” (Very natural) to “5” (Not natural at all).

Agustín Gravano - ACL - June Entrainment & Naturalness Measure of Entrainment Where fraction(w, S i )  Fraction of times Speaker i used word w in the conversation Examples entr(‘okay’)   | 10 / 500 – 8 / 600 |   entr(‘yeah’)   | 1 / 500 – 30 / 600 |   0.048

Agustín Gravano - ACL - June Entrainment & Naturalness Machine Learning Task Predict the perceived naturalness of conversations. Binary decision, over balanced data 250 conversations rated “1” (very natural), and 250 with ratings “3”, “4” or “5”. Computed entr(w) for the 100 most frequent words in the entire Switchboard corpus. Feature selection: 25 most predictive words. um, how, okay, go, I’ve, all, very, as, or, up, a, no, more, something, from, this, what, too, got, can, he, in, things, you, and.

Agustín Gravano - ACL - June Entrainment & Naturalness Results Logistic regression model (10-fold CV): 63.76% accuracy (significantly better than 50% baseline) Entrainment in usage of HFW is a good indicator of perceived naturalness.

Agustín Gravano - ACL - June Entrainment & Task Success Is a conversation more likely to succeed when HFW entrainment occurs? Columbia Games Corpus 12 spontaneous task-oriented dialogues in American English, with no eye contact. Each pair of subjects played a series of computer- based matching games. Subjects received a score after each task.

Agustín Gravano - ACL - June Entrainment & Task Success Measures of Entrainment Wherec= Class of words count Si (w) = No. of times S i used word w in the conversation

Agustín Gravano - ACL - June MF-G: 25 most frequent words in the game 25MF-C: 25 most frequent words in the corpus the, a, okay, and, of, I, on, right, is, it, that, have,… ACW: Affirmative cue words alright, mm-hm, okay, right, uh-huh, yeah, yes 7.9% of all words in the Games Corpus Entrainment & Task Success Word Classes

Agustín Gravano - ACL - June Correlations with game score: HFW entrainment positively correlated with task success. Entrainment & Task Success Results Word class ENTR 1 cor (p) ENTR 2 cor (p) 25MF-C0.341 (0.02)0.187 (0.20) 25MF-G0.376 (0.01)0.260 (0.07) ACW0.230 (0.12)0.372 (0.01)

Agustín Gravano - ACL - June Is dialogue more coordinated when HFW entrainment occurs? Columbia Games Corpus Labeled for type of turn exchanges (Beattie, 1982), including: Smooth Switch: S 2 starts his turn after S 1 has finished hers Interruption: S 2 starts his turn before S 1 has finished hers Overlap: S 2 starts his turn just before S 1 has finished hers, but without interrupting. Entrainment & Coordination

Agustín Gravano - ACL - June Significant correlations (p<0.05): ENTR 1 (ACW) & Prop. of Overlaps (cor = 0.64) ENTR 2 (ACW) & Prop. of Overlaps (cor = 0.61) ENTR 2 (25MF-G) & Prop. of Overlaps (cor = 0.60) ENTR 1 (25MF-C)& Prop. of Interruptions(cor = – 0.61) ENTR 2 (ACW) & Mean Latency of Smooth Switches (cor = – 0.76) HFW entrainment positively correlated with more overlaps, fewer interruptions, and shorter inter-turn latencies. Entrainment & Coordination Results

Agustín Gravano - ACL - June Conclusion Two novel measures of lexical entrainment, based on the usage of high-frequency words. Entrainment in usage of high-frequency words is correlated with: Perceived naturalness Task success Dialogue coordination Implications in the development of SDS.

High Frequency Word Entrainment in Spoken Dialogue Julia Ani Nenkova - Agustín Gravano - Julia Hirschberg ACL, June Columbus, OH Department of Computer and Information Science University of Pennsylvania - Philadelphia, PA Department of Computer Science Columbia University - New York, NY