Agustín Gravano1,2 Julia Hirschberg1 Turn-Yielding Cues in Task-Oriented Dialogue Agustín Gravano1,2 Julia Hirschberg1 Columbia University, New York, USA (2) Universidad de Buenos Aires, Argentina

Interactive Voice Response Systems Introduction Interactive Voice Response Systems Quickly spreading. “Uncomfortable”, “awkward”. ASR+TTS account for most IVR problems. Other problems revealed. Coordination of system-user exchanges. Long pauses after user turns; interruptions. Modeling turn-taking behavior should lead to improved system-user coordination. Begin to show  revealed An improved  improved Agustín Gravano SIGdial 2009

Introduction Goal Learn when the speaker is likely to end her/his conversational turn. Find turn-yielding cues. Cues displayed by the speaker when approaching a potential turn boundary. This should improve the coordination of IVRs: Speech understanding: Detect the end of the user’s turn. Speech generation: Display cues signalling the end of system’s turn. Could improve  should improve Agustín Gravano SIGdial 2009

Talk Outline Previous work Material Method Results Conclusions Rest of this talk  Talk Outline or Outline of Talk Agustín Gravano SIGdial 2009

Previous Work on Turn-Taking Duncan 1972, 1973, 1974, inter alia. Hypothesized 6 turn-yielding cues in face-to-face dialogue. Conjectured a linear relation between the number of displayed cues and the likelihood of a turn-taking attempt. Studies formalized and verified some of Duncan’s hypotheses. [For&Tho96; Wen&Sie03; Cut&Pea86; Wic&Cas01] Implementations of turn-boundary detection. Simulations [Fer&al.02,03; Edl&al.05; Sch06; Att&al.08; Bau08] Actual systems: Let’s Go! [Rau&Esk08] Exploiting turn-yielding cues improves performance. Same comment on duncan as for interspeech talk Agustín Gravano SIGdial 2009

Columbia Games Corpus 12 task-oriented spontaneous dialogues. Material Columbia Games Corpus 12 task-oriented spontaneous dialogues. Standard American English. 13 subjects: 6 female, 7 male. Series of collaborative computer games. No eye contact. No speech restrictions. 9 hours of dialogue. Manual orthographic transcription, alignment. Manual prosodic annotations (ToBI). Agustín Gravano SIGdial 2009

Columbia Games Corpus Material Player 1: Describer Player 2: Follower In an Objects games, each player saw a board with 5-7 objects. The boards were almost identical, with one object misplaced. One of the players had to describe the position of the target object to the other player, who had to move it to the correct position. Agustín Gravano SIGdial 2009

Turn-Yielding Cues Cues displayed by the speaker when approaching a potential turn boundary. Agustín Gravano SIGdial 2009

Method Turn-Yielding Cues IPU (Inter Pausal Unit): Maximal sequence of words from the same speaker surrounded by silence ≥ 50ms. Hold Smooth switch Speaker A: Speaker B: IPU1 IPU2 IPU3 Smooth switch: Speaker A finishes her utterance; speaker B takes the turn with no overlapping speech. Trained annotators distinguished Smooth switches from Interruptions and Backchannels using a scheme based on Ferguson 1977, Beattie 1982. Agustín Gravano SIGdial 2009

Method Turn-Yielding Cues To find turn-yielding cues, we compare: Speaker A: Speaker B: Hold Smooth switch IPU1 IPU2 IPU3 To find turn-yielding cues, we compare: IPUs preceding Holds, IPUs preceding Smooth switches. ~200 features: acoustic, prosodic, lexical, syntactic. Agustín Gravano SIGdial 2009

Individual Cues Final intonation: Faster speaking rate. Turn-Yielding Cues Individual Cues Final intonation: Falling (L-L%) or high-rising (H-H%). Faster speaking rate. Reduction of final lengthening. Lower intensity level. Lower pitch level. Higher jitter, shimmer, NHR. Related to perception of voice quality. Longer IPU duration (seconds and #words). Again, make it clear that you looked for many more…. Agustín Gravano SIGdial 2009

Before smooth switches: Turn-Yielding Cues Individual Cues Textual completion (independent of intonation). (1) Manually annotated a portion of the data. Labelers read up to the end of a target IPU (no right context), judged whether it could constitute a ‘complete’ utterance. 400 tokens. K=0.81. (2) Trained an SVM classifier. 19 lexical + syntactic features. Accuracy: 80%. Maj-class baseline: 55%. Human agreement: 91%. (3) Labeled all IPUs in the corpus with the SVM model. Again, make it clear that you looked for many more…. Incomplete Complete Before smooth switches: Before holds: 18% 82% 47% 53% (X2 test, p ~ 0) Agustín Gravano SIGdial 2009

Individual Cues Final intonation: L-L% or H-H%. Faster speaking rate. Turn-Yielding Cues Individual Cues Final intonation: L-L% or H-H%. Faster speaking rate. Lower intensity level. Lower pitch level. Higher jitter, shimmer, NHR. Longer IPU duration. Textual completion. Again, make it clear that you looked for many more…. Agustín Gravano SIGdial 2009

Defining Presence of a Cue Turn-Yielding Cues Defining Presence of a Cue 2-3 representative features for each cue: Final intonation Abs. pitch slope over final 200ms, 300ms. Speaking rate Syllables/sec, phonemes/sec over IPU. Intensity level Mean intensity over final 500ms, 1000ms. Pitch level Mean pitch over final 500ms, 1000ms. Voice quality Jitter, shimmer, NHR over final 500ms. IPU duration Duration in ms, and in number of words. Textual completion Complete vs. incomplete (binary). Define presence/absence based on whether the value is closer to the mean before S or H. Agustín Gravano SIGdial 2009

Top Frequencies of Complex Cues digit == cue present dot == cue absent Turn-yielding cues: 1: Final intonation 2: Speaking rate 3: Intensity level 4: Pitch level 5: IPU duration 6: Voice quality 7: Completion Agustín Gravano SIGdial 2009

Number of cues conjointly displayed Turn-Yielding Cues Combined Cues r 2 = 0.969 Percentage of turn-taking attempts Number of cues conjointly displayed Agustín Gravano SIGdial 2009

IVR Systems After each IPU from the user: Turn-Yielding Cues IVR Systems After each IPU from the user: if estimated likelihood > threshold then take the turn To signal the end of a system’s turn: Include as many cues as possible in the system’s final IPU. Agustín Gravano SIGdial 2009

Summary Study of turn-yielding cues. Objective, automatically computable. Combined cues. Improve turn-taking decisions of IVR systems. Results drawn from task-oriented dialogues. Not necessarily generalizable. Suitable for most IVR domains. Interspeech 2009: Study of backchannel-inviting cues. Agustín Gravano SIGdial 2009

Special thanks to… Julia Hirschberg Thesis Committee Members Maxine Eskenazi, Kathy McKeown, Becky Passonneau, Amanda Stent. Speech Lab at Columbia University Stefan Benus, Fadi Biadsy, Sasha Caskey, Bob Coyne, Frank Enos, Martin Jansche, Jackson Liscombe, Sameer Maskey, Andrew Rosenberg. Collaborators Gregory Ward and Elisa Sneed German (Northwestern U); Ani Nenkova (UPenn); Héctor Chávez, David Elson, Michel Galley, Enrique Henestroza, Hanae Koiso, Shira Mitchell, Michael Mulley, Kristen Parton, Ilia Vovsha, Lauren Wilcox. Agustín Gravano SIGdial 2009