Grounding by nodding GESPIN 2009, Poznan, Poland Volha Petukhova and Harry Bunt Tilburg Center for Creative Computing

Introduction Research questions Method Experiments Results Conclusions Content Introduction Research questions Method Experiments Results Conclusions

Introduction Research questions Method Experiments Results Conclusions The coordination of the beliefs and assumptions of the participants is a central issue in any communication A set of propositions that the dialogue participants mutually believe is called their common ground The process of establishing and updating the common ground is called grounding Introduction Research questions Method Experiments Results Conclusions

Introduction Research questions Method Experiments Results Conclusions ‘common ground’ is not directly observable grounding mechanisms are accessible through observable dialogue behavior, e.g. evidence of understanding what is said in dialogue is provided by feedback acts.

Contribution Model (Clark and Schaefer, 1989): Introduction Contribution Model (Clark and Schaefer, 1989): (1) content specification (a speaker tries to specify the content of his contribution, and the partners try to register that content), and (2) grounding (participants attempt to establish the mutual belief that they understand what was said). Each contribution has two phases: a presentation phase an acceptance phase Introduction Research questions Method Experiments Results Conclusions

Introduction Computational model of grounding (Traum, 1994) : Research questions Method Experiments Results Conclusions Computational model of grounding (Traum, 1994) : grounding acts that have a specific function in advancing the mutual understanding discourse units which consists of an initial presentation and as many utterances as needed to make this act mutually understood.

Introduction Research questions Method Experiments Results Conclusions Problems: Traum’s model consider grounding as the process of establishing the mutual understanding of each other’s intentions and actions, not that of the utterance content. Contribution Model requires that participants specify the content, but does not provide means to represent the content of contributions. Both models do not compute the semantic content of an utterance to specify what information is being or has to be grounded Introduction Research questions Method Experiments Results Conclusions

Introduction DIT model: - dialogue context - dialogue acts Dialogue acts are defined as operators that update contexts in certain ways, which can be described by the communicative function and the semantic content : Semantic Content (SC) : entities (objects, events, etc.) that the utterance is about; Communicative Function (CF) : the purpose that the utterance has in the communication; The CF describes how the SC is to be used to update the Information State (context) of the participants (cf. [Bunt, 1989]). Introduction Research questions Method Experiments Results Conclusions

Introduction Research questions Method Experiments Results Conclusions Information is transferred from one dialogue participant to another through belief creation (understanding) and belief transfer (adoption). For example: if A requests B to perform an action then the understanding of A’s request will be: B believes that A wants B to perform an action, and B believes that A assumes that B is able to perform this action. If B reacts as ‘Yes, of course’, then A may be expected to believe that B plans to perform the requested action (adoption). Introduction Research questions Method Experiments Results Conclusions

Introduction Research questions Method Experiments Results Conclusions Nonverbal means play an important role in the grounding process in face-to-face dialogue: eye gaze is the most basic form of showing attention (Goodwin, 1981; Novick et al. ,1996; Nakano et al., 2003) head nods signal that the previous utterance was understood (Clark 1996)

olga: Make chronically; blocks appear with B said something and so on (try at least) Then flying boxes with beliefs Example B: wording: But th gaze: person D posture: working position regardless we're we're aiming for the under sixty five person A person D person A personC personA I think person A A believes that she understood the B’s utterance correctly A believes that B wants to continues as a speaker A: wording: gaze: person B head posture working position A believes that B wants to have the turn Under sixty five okay That’s a good start short multiple nods(4) table bowing A reports about the positive evaluation of B’s utterance and offers for further debate short single nod short multiple nods(5) D: Wording: Gaze: person A Head: Posture: working position table person B short multiple nods(5) C believes that B believes that p and C believes that p is true D believes that B wants to continues as a speaker C: Wording: Gaze: person D Head: Face: Posture: working position Yep long nods(2) blinking person A person B person A

Research Question which types of nonverbal expressions and their combinations can be interpreted as adoption signals and which merely signal understanding?

Introduction Research questions Method Experiments Results Conclusions Corpus material AMI-meetings (http://www.amiproject.org/) Two scenario-based meetings selected with a total duration of 51 minutes (English native speakers) Introduction Research questions Method Experiments Results Conclusions

Introduction Research questions Method Experiments Results Conclusions Transcriptions Verbal elements: manually produced orthographic transcriptions for each speaker, including word-level timings Non-verbal elements: gaze direction; head movements; hand and arm gestures; eyebrow, eyes and lips movements; posture shifts; features: form of movement (head: nod, shake, jerk; hands: pointing, shoulder-shrug, etc.; eyes: narrow, widen; lips: pout, compress, purse, flatten, (half)open, random moves); direction (up, down, left, right, backward, forward); trajectory (line, circle, arch); size (large, small, medium, extra large); speed (slow, medium, fast); number of repetitions (up to 20 times); FTO: difference between time that turn starts and moment that previous turn ends; duration. Introduction Research questions Method Experiments Results Conclusions

Introduction Research questions Method Experiments Results Conclusions Annotations Speech and nonverbal signs annotated per dimension with the DIT++ tagset (http://dit.uvt.nl/) using ANVIL Segmentation performed per dimension into functional segments (Geertzen et al., 2007) Introduction Research questions Method Experiments Results Conclusions

Introduction Research questions Method Experiments Results Conclusions Perception experiments: outline 60 video clips with 6 different speakers (3 male, 3 female; English native speakers) duration of each clip was about 10 seconds and contained the full turns of the previous speaker and the current speaker 16 naïve subjects (4 male and 12 female, all between the ages of 20 and 40) Task: to answer the question whether they think that a participant understands the dialogue act of the previous speaker or that he/she agrees with the previous speaker. 10 seconds to react to each stimulus and were allowed to watch every video as many times as they liked Introduction Research questions Method Experiments Results Conclusions

Examples:

Introduction Research questions Method Experiments Results Conclusions Perception experiments: inter-subject agreement Introduction Research questions Method Experiments Results Conclusions Task Kappa `Understanding’ label 0.54 `Adoption’ label 0.9 Overall 0.68

Introduction Research questions Method Experiments Results Conclusions Perception experiments: results Feature Pearson’s R head nod(-s) + wording .55* (p=0.000) head nod(-s) + ‘yeah’ .43* (p=0.000) floor transfer offset .34* (p=0.07) number of repetitions .25* (p=0.045) head nod .29* (p=0.02) head jerk -.29* (p=0.02) gaze pattern ‘person-averted’ .47* (p=0.06) head nod(-s) + blinking .25* (p=0.49) head nod(-s) + lips movements .42* (p=0.001) slow multiple nods .37* (p=0.003)     Introduction Research questions Method Experiments Results Conclusions head nod(-s) + ‘uh-uhu’ .2 (p=0.123) duration .17 (p=0.186) speed of movements .22 (p=0.07) size of movements .027 (p=0.834) head nod(-s) + eyebrows movement .012 (p=0.925) head nod(-s) + hand movements .039 (p=0.762) head nod(-s) + posture shift -.16 (p=0.210) fast single nod -.13 (p=0.305) fast multiple nods .13 (p=0.32) slow single nod -.025 (p=0.847)        

Introduction Research questions Method Experiments Results Conclusions Perception experiments: results Feature Pearson’s R head nod(-s) + wording .55* (p=0.000) head nod(-s) + ‘yeah’ .43* (p=0.000) floor transfer offset .34* (p=0.07) number of repetitions .25* (p=0.045) head nod .29* (p=0.02) head jerk -.29* (p=0.02) gaze pattern ‘person-averted’ .47* (p=0.06) head nod(-s) + blinking .25* (p=0.49) head nod(-s) + lips movements .42* (p=0.001) slow multiple nods .37* (p=0.003)     Introduction Research questions Method Experiments Results Conclusions        

Introduction Research questions Method Experiments Results Conclusions Speakers provide different types of evidence for understanding and adoption Nodding is not enough for successful interpretation Intention is recognized more successfully and accurately if rely on the information from multiple signals and modalities Introduction Research questions Method Experiments Results Conclusions

Introduction Research questions Method Experiments Results Conclusions Future work Grounding costs when participants have no direct eye contact (for negotiation tasks, problem-solving, or non-collaborative tasks) Grounding costs in situations with severe limitations on the use of speech Introduction Research questions Method Experiments Results Conclusions

Thank you! ?