On the role of context and prosody in the interpretation of ‘okay’ Julia Agustín Gravano, Stefan Benus, Julia Hirschberg Héctor Chávez, and Lauren Wilcox ACL, June 2007, Prague Spoken Language Processing Group Columbia University

Agustín Gravano - ACL - June Overview What information do subjects use to interpret the word ‘okay’ in dialogue? Perception Study. Findings: Contextual cues stronger predictors than acoustic / prosodic / phonetic cues. Final rising pitch: Strongest prosodic cue.

Agustín Gravano - ACL - June Overview that’s pretty much okay Speaker 1: between the yellow mermaid and the whale Speaker 2: okay Speaker 1: and it is okay we gonna be placing the blue moon

Agustín Gravano - ACL - June Cue Words Linguistic expressions that can be used to convey information about the discourse structure, or to make a semantic contribution. Discourse markers, cue phrases, clue words, … Examples: now, well, so, alright, and, okay, first, on the other hand, by the way, for example, …

Agustín Gravano - ACL - June Research Questions In spoken dialogue, how do hearers disambiguate cue words? How important is acoustic/prosodic information? What is the role of phonetic variation? What is the role of discourse context?

Agustín Gravano - ACL - June Why do we care? Spoken dialogue systems Need to convey potentially ambiguous terms with a particular intended meaning. Must interpret the user’s input correctly.

Agustín Gravano - ACL - June Previous Work Cues to cue phrase disambiguation Hirschberg & Litman ’87, ’93; Hockey ’93; Litman ’94 Cues to Dialogue Act identification Jurafsky et al ’98; Rosset & Lamel ’04 Contextual cues to the production of backchannels Ward & Tsukahara ’00; Sanjanhar & Ward ’06

Agustín Gravano - ACL - June The Columbia Games Corpus 12 spontaneous task-oriented dyadic conversations in Standard American English. 2 subjects playing a computer game, no eye contact. Describer: Follower:

Agustín Gravano - ACL - June The Columbia Games Corpus Annotation of Affirmative Cue Words Cue Words alright gotcha huh mm-hm okay right uh-huh yeah yep yes yup Functions Acknowledgment / Agreement Backchannel Cue beginning discourse segment Cue ending discourse segment Check with the interlocutor Stall / Filler Back from a task Literal modifier Pivot beginning Pivot ending count 1. the of okay and like 753 …

Agustín Gravano - ACL - June Perception Study Experiment Design okay Speaker 1: but it's gonna be below the onion Speaker 2: okay Cue beginning discourse segment Backchannel Acknowledgment / Agreement Speaker 1: okay alright I'll try it okay Speaker 2: okay the owl is blinking Speaker 1: yeah um there's like there's some space there's Speaker 2: okay I think I got it

Agustín Gravano - ACL - June contextualized ‘okay’ Perception Study Experiment Design 54 instances of ‘okay’ (18 for each function). 2 tokens for each ‘okay’:  Isolated condition: Only the word ‘okay’.  Contextualized condition: 2 full speaker turns: The turn containing the target ‘okay’; and The previous turn by the other speaker. speakersokay

Agustín Gravano - ACL - June Perception Study Experiment Design Two parts: Part 1: 54 isolated tokens Part 2: 54 contextualized tokens Subjects asked to classify each token of ‘okay’ as: Acknowledgment / Agreement, or Backchannel, or Cue beginning discourse segment.

Agustín Gravano - ACL - June Perception Study Experiment Implementation Subjects: 20 paid subjects (10 female, 10 male). Ages between 20 and 60. Native speakers of English. No hearing problems. GUI on a laboratory workstation with headphones.

Agustín Gravano - ACL - June Results Inter-Subject Agreement Kappa measure of agreement with respect to chance (Fleiss ’71) Isolated ConditionContextualized Condition Overall Ack / Agree vs. Other Backchannel vs. Other Cue beginning vs. Other

Agustín Gravano - ACL - June Results Cues to Interpretation Phonetic transcription of okay: Isolated Condition Strong correlation for realization of  :  Backchannel  Ack/Agree, Cue Beginning Contextualized Condition No strong correlations found for phonetic variants.

Agustín Gravano - ACL - June Results Cues to Interpretation Isolated ConditionContextualized Condition Ack / Agree Shorter /k/Shorter latency between turns Shorter pause before okay Backchannel Lower intensity Higher final pitch slope Longer 2 nd syllable Higher final pitch slope More words by S2 before okay Fewer words by S1 after okay Cue beginning Lower final pitch slope Lower overall pitch slope Longer latency between turns More words by S1 after okay Lower final pitch slope S1 = Utterer of the target ‘okay’. S2 = The other speaker.

Agustín Gravano - ACL - June Final intonation using the ToBI conventions. (Both isolated and contextualized conditions.) H-H%  Backchannel H-L% L-H%  Ack/Agree, Backchannel L-L%  Ack/Agree, Cue beginning Results Cues to Interpretation

Agustín Gravano - ACL - June Conclusions Agreement: Availability of context improves inter-subject agreement. Cue beginnings easier to disambiguate than the other two functions. Cues to interpretation: Contextual features trump features of word okay. Exception: Final pitch slope of okay in both conditions.

Agustín Gravano - ACL - June Further Work Benus et al, 2007 “The prosody of backchannels in American English”, ICPhS 2007, Saarbrücken, Germany, August Gravano et al, 2007 “Classification of discourse functions of affirmative words in spoken dialogue”, Interspeech 2007, Antwerp, Belgium, August 2007.

On the role of context and prosody in the interpretation of ‘okay’ Julia Agustín Gravano, Stefan Benus, Julia Hirschberg Héctor Chávez, and Lauren Wilcox ACL, June 2007, Prague Spoken Language Processing Group Columbia University

Agustín Gravano - ACL - June The Columbia Games Corpus Annotation Orthographic transcription and alignment. Laughs, coughs, breaths, smacks, throat-clearings. Self repairs. Intonation, using the ToBI convention. Function of affirmative cue words (alright, mm-hm, okay, right, uh-huh, yeah, yes, …). Question form and function.

Agustín Gravano - ACL - June The Columbia Games Corpus Annotation of Intonation: ToBI Tones: Pitch accents:L*, H*, L*+H, H+!H*, … Phrase accents:L-, H-, !H- Boundary tones:L%, H% Break Indices: Degrees of junction 0 = no word boundary... 4 = full intonational phrase boundary Miscellanea: Disfluencies, non-speech sounds, etc.

Agustín Gravano - ACL - June The Columbia Games Corpus Annotation of Intonation: ToBI waveform fundamental frequency (F0)

Agustín Gravano - ACL - June Acknowledge/Agreement: The function of ‘okay’ that indicates “I believe what you said” and/or “I agree with what you say”. Backchannel: The function of ‘okay’ in response to another speaker's utterance that indicates only “I’m still here” or “I hear you and please continue”. Cue beginning discourse segment The function of ‘okay’ that marks a new segment of a discourse or a new topic. This use of ‘okay’ could be replaced by ‘now’. Perception Study Definitions Given to the Subjects