2. On the role of context and prosody in the interpretation of ‘okay’ Julia Agustín Gravano, Stefan Benus, Julia Hirschberg Héctor Chávez, and Lauren Wilcox ACL, June 2007, Prague Spoken Language Processing Group Columbia University

3. Agustín Gravano - ACL - June 20072 Overview What information do subjects use to interpret the word ‘okay’ in dialogue? Perception Study. Findings: Contextual cues stronger predictors than acoustic / prosodic / phonetic cues. Final rising pitch: Strongest prosodic cue.

4. Agustín Gravano - ACL - June 20073 Overview that’s pretty much okay Speaker 1: between the yellow mermaid and the whale Speaker 2: okay Speaker 1: and it is okay we gonna be placing the blue moon

5. Agustín Gravano - ACL - June 20074 Cue Words Linguistic expressions that can be used to convey information about the discourse structure, or to make a semantic contribution. Discourse markers, cue phrases, clue words, … Examples: now, well, so, alright, and, okay, first, on the other hand, by the way, for example, …

6. Agustín Gravano - ACL - June 20075 Research Questions In spoken dialogue, how do hearers disambiguate cue words? How important is acoustic/prosodic information? What is the role of phonetic variation? What is the role of discourse context?

7. Agustín Gravano - ACL - June 20076 Why do we care? Spoken dialogue systems Need to convey potentially ambiguous terms with a particular intended meaning. Must interpret the user’s input correctly.

8. Agustín Gravano - ACL - June 20077 Previous Work Cues to cue phrase disambiguation Hirschberg & Litman ’87, ’93; Hockey ’93; Litman ’94 Cues to Dialogue Act identification Jurafsky et al ’98; Rosset & Lamel ’04 Contextual cues to the production of backchannels Ward & Tsukahara ’00; Sanjanhar & Ward ’06

9. Agustín Gravano - ACL - June 20078 The Columbia Games Corpus 12 spontaneous task-oriented dyadic conversations in Standard American English. 2 subjects playing a computer game, no eye contact. Describer: Follower:

10. Agustín Gravano - ACL - June 20079 The Columbia Games Corpus Annotation of Affirmative Cue Words Cue Words alright gotcha huh mm-hm okay right uh-huh yeah yep yes yup Functions Acknowledgment / Agreement Backchannel Cue beginning discourse segment Cue ending discourse segment Check with the interlocutor Stall / Filler Back from a task Literal modifier Pivot beginning Pivot ending count 1. the4565 2. of 1534 3. okay1151 4. and 886 5. like 753 …

11. Agustín Gravano - ACL - June 200710 Perception Study Experiment Design okay Speaker 1: but it's gonna be below the onion Speaker 2: okay Cue beginning discourse segment Backchannel Acknowledgment / Agreement Speaker 1: okay alright I'll try it okay Speaker 2: okay the owl is blinking Speaker 1: yeah um there's like there's some space there's Speaker 2: okay I think I got it

12. Agustín Gravano - ACL - June 200711 contextualized ‘okay’ Perception Study Experiment Design 54 instances of ‘okay’ (18 for each function). 2 tokens for each ‘okay’:  Isolated condition: Only the word ‘okay’.  Contextualized condition: 2 full speaker turns: The turn containing the target ‘okay’; and The previous turn by the other speaker. speakersokay

13. Agustín Gravano - ACL - June 200712 Perception Study Experiment Design Two parts: Part 1: 54 isolated tokens Part 2: 54 contextualized tokens Subjects asked to classify each token of ‘okay’ as: Acknowledgment / Agreement, or Backchannel, or Cue beginning discourse segment.

14. Agustín Gravano - ACL - June 200713 Perception Study Experiment Implementation Subjects: 20 paid subjects (10 female, 10 male). Ages between 20 and 60. Native speakers of English. No hearing problems. GUI on a laboratory workstation with headphones.

15. Agustín Gravano - ACL - June 200714 Results Inter-Subject Agreement Kappa measure of agreement with respect to chance (Fleiss ’71) Isolated ConditionContextualized Condition Overall.120.294 Ack / Agree vs. Other.089.227 Backchannel vs. Other.118.164 Cue beginning vs. Other.157.497

16. Agustín Gravano - ACL - June 200715 Results Cues to Interpretation Phonetic transcription of okay: Isolated Condition Strong correlation for realization of  :  Backchannel  Ack/Agree, Cue Beginning Contextualized Condition No strong correlations found for phonetic variants.

17. Agustín Gravano - ACL - June 200716 Results Cues to Interpretation Isolated ConditionContextualized Condition Ack / Agree Shorter /k/Shorter latency between turns Shorter pause before okay Backchannel Lower intensity Higher final pitch slope Longer 2 nd syllable Higher final pitch slope More words by S2 before okay Fewer words by S1 after okay Cue beginning Lower final pitch slope Lower overall pitch slope Longer latency between turns More words by S1 after okay Lower final pitch slope S1 = Utterer of the target ‘okay’. S2 = The other speaker.

18. Agustín Gravano - ACL - June 200717 Final intonation using the ToBI conventions. (Both isolated and contextualized conditions.) H-H%  Backchannel H-L% L-H%  Ack/Agree, Backchannel L-L%  Ack/Agree, Cue beginning Results Cues to Interpretation

19. Agustín Gravano - ACL - June 200718 Conclusions Agreement: Availability of context improves inter-subject agreement. Cue beginnings easier to disambiguate than the other two functions. Cues to interpretation: Contextual features trump features of word okay. Exception: Final pitch slope of okay in both conditions.

20. Agustín Gravano - ACL - June 200719 Further Work Benus et al, 2007 “The prosody of backchannels in American English”, ICPhS 2007, Saarbrücken, Germany, August 2007. Gravano et al, 2007 “Classification of discourse functions of affirmative words in spoken dialogue”, Interspeech 2007, Antwerp, Belgium, August 2007.

21. On the role of context and prosody in the interpretation of ‘okay’ Julia Agustín Gravano, Stefan Benus, Julia Hirschberg Héctor Chávez, and Lauren Wilcox ACL, June 2007, Prague Spoken Language Processing Group Columbia University

22. Agustín Gravano - ACL - June 200721 The Columbia Games Corpus Annotation Orthographic transcription and alignment. Laughs, coughs, breaths, smacks, throat-clearings. Self repairs. Intonation, using the ToBI convention. Function of affirmative cue words (alright, mm-hm, okay, right, uh-huh, yeah, yes, …). Question form and function.

23. Agustín Gravano - ACL - June 200722 The Columbia Games Corpus Annotation of Intonation: ToBI Tones: Pitch accents:L*, H*, L*+H, H+!H*, … Phrase accents:L-, H-, !H- Boundary tones:L%, H% Break Indices: Degrees of junction 0 = no word boundary... 4 = full intonational phrase boundary Miscellanea: Disfluencies, non-speech sounds, etc.

24. Agustín Gravano - ACL - June 200723 The Columbia Games Corpus Annotation of Intonation: ToBI waveform fundamental frequency (F0)

25. Agustín Gravano - ACL - June 200724 Acknowledge/Agreement: The function of ‘okay’ that indicates “I believe what you said” and/or “I agree with what you say”. Backchannel: The function of ‘okay’ in response to another speaker's utterance that indicates only “I’m still here” or “I hear you and please continue”. Cue beginning discourse segment The function of ‘okay’ that marks a new segment of a discourse or a new topic. This use of ‘okay’ could be replaced by ‘now’. Perception Study Definitions Given to the Subjects