1. High Frequency Word Entrainment in Spoken Dialogue ACL, June 2008 - Columbus, OH Department of Computer and Information Science University of Pennsylvania - Philadelphia, PA Department of Computer Science Columbia University - New York, NY Julia Ani Nenkova - Agustín Gravano - Julia Hirschberg

2. Agustín Gravano - ACL - June 20082 Entrainment In conversation, people adapt the way they speak to match their partners’. Entrainment, accommodation, adaptation, alignment, convergence.

3. Agustín Gravano - ACL - June 20083 Previous Work Existence of entrainment In conversation, speakers: Negotiate common ways of describing things. S.E. Brennan, 1996 Alter their intensity to match their partners’. R. Coulston et al., 2002 A. Ward & D. Litman, 2007 Reuse syntactic constructions. D. Reitter et al., 2006

4. Agustín Gravano - ACL - June 20084 Previous Work Role of entrainment Entrainment at different levels (lex, syn, sem): Is key for both production and understanding, and facilitates interaction. M.J. Pickering & S. Garrod, 2004 D. Goleman, 2006 Is a good predictor of task success (MapTask). D. Reitter & J. Moore, 2007

5. Agustín Gravano - ACL - June 20085 This Work Novel measures of entrainment based on usage of high-frequency words (HFW). Entrainment and… Perceived naturalness Task success Dialogue coordination Implications in the development of Spoken Dialogue Systems.

6. Agustín Gravano - ACL - June 20086 High-Frequency Words Most common words in a corpus, or in a conversation. Typically, function words and cue words. Entrainment of HFW Domain-independent

7. Agustín Gravano - ACL - June 20087 Entrainment & Naturalness Will a conversation be perceived as more natural if HFW entrainment occurs? Switchboard corpus 2430 spontaneous telephone conversations in American English Speakers asked to discuss a pre-assigned topic Annotated for degree of perceived naturalness, from “1” (Very natural) to “5” (Not natural at all).

8. Agustín Gravano - ACL - June 20088 Entrainment & Naturalness Measure of Entrainment Where fraction(w, S i )  Fraction of times Speaker i used word w in the conversation Examples entr(‘okay’)   | 10 / 500 – 8 / 600 |   0.0067 entr(‘yeah’)   | 1 / 500 – 30 / 600 |   0.048

9. Agustín Gravano - ACL - June 20089 Entrainment & Naturalness Machine Learning Task Predict the perceived naturalness of conversations. Binary decision, over balanced data 250 conversations rated “1” (very natural), and 250 with ratings “3”, “4” or “5”. Computed entr(w) for the 100 most frequent words in the entire Switchboard corpus. Feature selection: 25 most predictive words. um, how, okay, go, I’ve, all, very, as, or, up, a, no, more, something, from, this, what, too, got, can, he, in, things, you, and.

10. Agustín Gravano - ACL - June 200810 Entrainment & Naturalness Results Logistic regression model (10-fold CV): 63.76% accuracy (significantly better than 50% baseline) Entrainment in usage of HFW is a good indicator of perceived naturalness.

11. Agustín Gravano - ACL - June 200811 Entrainment & Task Success Is a conversation more likely to succeed when HFW entrainment occurs? Columbia Games Corpus 12 spontaneous task-oriented dialogues in American English, with no eye contact. Each pair of subjects played a series of computer- based matching games. Subjects received a score after each task.

12. Agustín Gravano - ACL - June 200812 Entrainment & Task Success Measures of Entrainment Wherec= Class of words count Si (w) = No. of times S i used word w in the conversation

13. Agustín Gravano - ACL - June 200813 25MF-G: 25 most frequent words in the game 25MF-C: 25 most frequent words in the corpus the, a, okay, and, of, I, on, right, is, it, that, have,… ACW: Affirmative cue words alright, mm-hm, okay, right, uh-huh, yeah, yes 7.9% of all words in the Games Corpus Entrainment & Task Success Word Classes

14. Agustín Gravano - ACL - June 200814 Correlations with game score: HFW entrainment positively correlated with task success. Entrainment & Task Success Results Word class ENTR 1 cor (p) ENTR 2 cor (p) 25MF-C0.341 (0.02)0.187 (0.20) 25MF-G0.376 (0.01)0.260 (0.07) ACW0.230 (0.12)0.372 (0.01)

15. Agustín Gravano - ACL - June 200815 Is dialogue more coordinated when HFW entrainment occurs? Columbia Games Corpus Labeled for type of turn exchanges (Beattie, 1982), including: Smooth Switch: S 2 starts his turn after S 1 has finished hers Interruption: S 2 starts his turn before S 1 has finished hers Overlap: S 2 starts his turn just before S 1 has finished hers, but without interrupting. Entrainment & Coordination

16. Agustín Gravano - ACL - June 200816 Significant correlations (p<0.05): ENTR 1 (ACW) & Prop. of Overlaps (cor = 0.64) ENTR 2 (ACW) & Prop. of Overlaps (cor = 0.61) ENTR 2 (25MF-G) & Prop. of Overlaps (cor = 0.60) ENTR 1 (25MF-C)& Prop. of Interruptions(cor = – 0.61) ENTR 2 (ACW) & Mean Latency of Smooth Switches (cor = – 0.76) HFW entrainment positively correlated with more overlaps, fewer interruptions, and shorter inter-turn latencies. Entrainment & Coordination Results

17. Agustín Gravano - ACL - June 200817 Conclusion Two novel measures of lexical entrainment, based on the usage of high-frequency words. Entrainment in usage of high-frequency words is correlated with: Perceived naturalness Task success Dialogue coordination Implications in the development of SDS.

18. High Frequency Word Entrainment in Spoken Dialogue Julia Ani Nenkova - Agustín Gravano - Julia Hirschberg ACL, June 2008 - Columbus, OH Department of Computer and Information Science University of Pennsylvania - Philadelphia, PA Department of Computer Science Columbia University - New York, NY