1. Grounding by nodding GESPIN 2009, Poznan, Poland
Volha Petukhova and Harry Bunt
Tilburg Center for Creative Computing

2. Introduction Research questions Method Experiments Results Conclusions
Content
Introduction
Research questions
Method
Experiments
Results
Conclusions

3. Introduction Research questions Method Experiments Results Conclusions
The coordination of the beliefs and assumptions of the participants is a central issue in any communication
A set of propositions that the dialogue participants mutually believe is called their common ground
The process of establishing and updating the common ground is called grounding
Introduction
Research questions
Method
Experiments
Results
Conclusions

4. Introduction Research questions Method Experiments Results Conclusions
‘common ground’ is not directly observable
grounding mechanisms are accessible through observable dialogue behavior, e.g. evidence of understanding what is said in dialogue is provided by feedback acts.

5. Contribution Model (Clark and Schaefer, 1989):
Introduction
Contribution Model (Clark and Schaefer, 1989):
(1) content specification (a speaker tries to specify the content of his contribution, and the partners try to register that content), and
(2) grounding (participants attempt to establish the mutual belief that they understand what was said).
Each contribution has two phases:
a presentation phase
an acceptance phase
Introduction
Research questions
Method
Experiments
Results
Conclusions

6. Introduction Computational model of grounding (Traum, 1994) :
Research questions
Method
Experiments
Results
Conclusions
Computational model of grounding (Traum, 1994) :
grounding acts that have a specific function in advancing the mutual understanding
discourse units which consists of an initial presentation and as many utterances as needed to make this act mutually understood.

7. Introduction Research questions Method Experiments Results Conclusions
Problems:
Traum’s model consider grounding as the process of establishing the mutual understanding of each other’s intentions and actions, not that of the utterance content.
Contribution Model requires that participants specify the content, but does not provide means to represent the content of contributions.
Both models do not compute the semantic content of an utterance to specify what information is being or has to be grounded
Introduction
Research questions
Method
Experiments
Results
Conclusions

8. Introduction DIT model: - dialogue context - dialogue acts
Dialogue acts are defined as operators that update contexts in certain ways, which can be described by the communicative function and the semantic content :
Semantic Content (SC) : entities (objects, events, etc.) that the utterance is about;
Communicative Function (CF) : the purpose that the utterance has in the communication;
The CF describes how the SC is to be used to update the Information State (context) of the participants (cf. [Bunt, 1989]).
Introduction
Research questions
Method
Experiments
Results
Conclusions

9. Introduction Research questions Method Experiments Results Conclusions
Information is transferred from one dialogue participant to another through belief creation (understanding) and belief transfer (adoption).
For example:
if A requests B to perform an action then the understanding of A’s request will be:
B believes that A wants B to perform an action, and
B believes that A assumes that B is able to perform this action.
If B reacts as ‘Yes, of course’, then A may be expected to believe that B plans to perform the requested action (adoption).
Introduction
Research questions
Method
Experiments
Results
Conclusions

10. Introduction Research questions Method Experiments Results Conclusions
Nonverbal means play an important role in the grounding process in face-to-face dialogue:
eye gaze is the most basic form of showing attention (Goodwin, 1981; Novick et al. ,1996; Nakano et al., 2003)
head nods signal that the previous utterance was understood (Clark 1996)

11. olga:
Make chronically; blocks appear with B said something and so on (try at least)
Then flying boxes with beliefs
Example B:
wording: But th
gaze: person D posture: working position
regardless we're we're aiming for the under sixty five
person A person D person A personC personA
I think
person A
A believes that she understood the B’s utterance correctly
A believes that B wants to continues as a speaker
A: wording:
gaze: person B
head posture working position
A believes that B wants to have the turn
Under sixty five okay That’s a good start
short multiple nods(4)
table
bowing
A reports about the positive evaluation of B’s utterance and offers for further debate
short single nod
short multiple nods(5) D:
Wording:
Gaze: person A
Head:
Posture: working position
table
person B
short multiple nods(5)
C believes that B believes that p and C believes that p is true
D believes that B wants to continues as a speaker C:
Wording:
Gaze: person D
Head:
Face:
Posture: working position
Yep
long nods(2)
blinking
person A
person B
person A

12. Research Question
which types of nonverbal expressions and their combinations can be interpreted as adoption signals and which merely signal understanding?

13. Introduction Research questions Method Experiments Results Conclusions
Corpus material
AMI-meetings (
Two scenario-based meetings selected with a total duration of 51 minutes (English native speakers)
Introduction
Research questions
Method
Experiments
Results
Conclusions

14. Introduction Research questions Method Experiments Results Conclusions
Transcriptions
Verbal elements: manually produced orthographic transcriptions for each speaker, including word-level timings
Non-verbal elements: gaze direction; head movements; hand and arm gestures; eyebrow, eyes and lips movements; posture shifts; features:
form of movement (head: nod, shake, jerk; hands: pointing, shoulder-shrug, etc.; eyes: narrow, widen; lips: pout, compress, purse, flatten, (half)open, random moves);
direction (up, down, left, right, backward, forward);
trajectory (line, circle, arch);
size (large, small, medium, extra large);
speed (slow, medium, fast);
number of repetitions (up to 20 times);
FTO: difference between time that turn starts and moment that previous turn ends;
duration.
Introduction
Research questions
Method
Experiments
Results
Conclusions

15. Introduction Research questions Method Experiments Results Conclusions
Annotations
Speech and nonverbal signs annotated per dimension with the DIT++ tagset ( using ANVIL
Segmentation performed per dimension into functional segments (Geertzen et al., 2007)
Introduction
Research questions
Method
Experiments
Results
Conclusions

16. Introduction Research questions Method Experiments Results Conclusions
Perception experiments: outline
60 video clips with 6 different speakers (3 male, 3 female; English native speakers)
duration of each clip was about 10 seconds and contained the full turns of the previous speaker and the current speaker
16 naïve subjects (4 male and 12 female, all between the ages of 20 and 40)
Task: to answer the question whether they think that a participant understands the dialogue act of the previous speaker or that he/she agrees with the previous speaker.
10 seconds to react to each stimulus and were allowed to watch every video as many times as they liked
Introduction
Research questions
Method
Experiments
Results
Conclusions

17. Examples:

18. Introduction Research questions Method Experiments Results Conclusions
Perception experiments: inter-subject agreement
Introduction
Research questions
Method
Experiments
Results
Conclusions
Task
Kappa
`Understanding’ label
0.54
`Adoption’ label
0.9
Overall
0.68

19. Introduction Research questions Method Experiments Results Conclusions
Perception experiments: results
Feature
Pearson’s R
head nod(-s) + wording
.55* (p=0.000)
head nod(-s) + ‘yeah’
.43* (p=0.000)
floor transfer offset
.34* (p=0.07)
number of repetitions
.25* (p=0.045)
head nod
.29* (p=0.02)
head jerk
-.29* (p=0.02)
gaze pattern ‘person-averted’
.47* (p=0.06)
head nod(-s) + blinking
.25* (p=0.49)
head nod(-s) + lips movements
.42* (p=0.001)
slow multiple nods
.37* (p=0.003)
Introduction
Research questions
Method
Experiments
Results
Conclusions
head nod(-s) + ‘uh-uhu’
.2 (p=0.123)
duration
.17 (p=0.186)
speed of movements
.22 (p=0.07)
size of movements
.027 (p=0.834)
head nod(-s) + eyebrows movement
.012 (p=0.925)
head nod(-s) + hand movements
.039 (p=0.762)
head nod(-s) + posture shift
-.16 (p=0.210)
fast single nod
-.13 (p=0.305)
fast multiple nods
.13 (p=0.32)
slow single nod
-.025 (p=0.847)

20. Introduction Research questions Method Experiments Results Conclusions
Perception experiments: results
Feature
Pearson’s R
head nod(-s) + wording
.55* (p=0.000)
head nod(-s) + ‘yeah’
.43* (p=0.000)
floor transfer offset
.34* (p=0.07)
number of repetitions
.25* (p=0.045)
head nod
.29* (p=0.02)
head jerk
-.29* (p=0.02)
gaze pattern ‘person-averted’
.47* (p=0.06)
head nod(-s) + blinking
.25* (p=0.49)
head nod(-s) + lips movements
.42* (p=0.001)
slow multiple nods
.37* (p=0.003)
Introduction
Research questions
Method
Experiments
Results
Conclusions

21. Introduction Research questions Method Experiments Results Conclusions
Speakers provide different types of evidence for understanding and adoption
Nodding is not enough for successful interpretation
Intention is recognized more successfully and accurately if rely on the information from multiple signals and modalities
Introduction
Research questions
Method
Experiments
Results
Conclusions

22. Introduction Research questions Method Experiments Results Conclusions
Future work
Grounding costs when participants have no direct eye contact (for negotiation tasks, problem-solving, or non-collaborative tasks)
Grounding costs in situations with severe limitations on the use of speech
Introduction
Research questions
Method
Experiments
Results
Conclusions

23. Thank you! ?