Conversational role assignment problem in multi-party dialogues Natasa Jovanovic Dennis Reidsma Rutger Rienks TKI group University of Twente

Outline Research tasks at TKI Research tasks at TKI  Interpretation of multimodal human- human communication in the meetings Conversational Role Assignment Problem (CRAP) Conversational Role Assignment Problem (CRAP)  Towards automatic addressee detection

A framework for multimodal interaction research

Multimodal annotation tool

Who is talking to whom? CRAP as one of the main issues in multi- parity conversation (Traum 2003.) CRAP as one of the main issues in multi- parity conversation (Traum 2003.) Taxonomy of conversational roles (Herbert K. Clark) Taxonomy of conversational roles (Herbert K. Clark) speakeraddressee side participant all participants bystander eavesdropper all listener

Our goal: Our goal:  Automatic addressee identification in small group discussions Addressees in meeting conversations: single participant, group of people, whole audience Addressees in meeting conversations: single participant, group of people, whole audience Importance of the issue of addressing in multi-party dialogues Importance of the issue of addressing in multi-party dialogues

Addressing mechanisms What are relevant sources of information for addressee identification in the face-to-face meeting conversations? What are relevant sources of information for addressee identification in the face-to-face meeting conversations? How does the speaker express who is the addressee of his utterance? How does the speaker express who is the addressee of his utterance? How can we combine all this information in order to determine the addressee of the utterance? How can we combine all this information in order to determine the addressee of the utterance?

Sources of information Speech Speech  Linguistic markers  word classes: personal pronouns, determiners in combination with personal pronouns, possessive pronouns and adjectives, indefinite pronouns, etc.  Name detection ( vocatives)  Dialogue acts Gaze direction Gaze direction Pointing gestures Pointing gestures Context categories(features) Context categories(features)

Dialogue Acts and Addressee detection (I) How many addresses may have an utterance? How many addresses may have an utterance? According to dialog act theory an utterance or an utterance segment may have more than one conversational function. According to dialog act theory an utterance or an utterance segment may have more than one conversational function. Each DA has a addressee ==> an utterance may have several addresses Each DA has a addressee ==> an utterance may have several addresses

Dialogue Acts and Addressee detection (II) MRDA (Meeting Recorder Dialogue Acts)– tag set for labeling multiparty face to face meetings (ICSI) MRDA (Meeting Recorder Dialogue Acts)– tag set for labeling multiparty face to face meetings (ICSI) We use a huge subset of the MRDA set which is organized on two levels: We use a huge subset of the MRDA set which is organized on two levels:  Forward looking functions (FLF )  Backward looking functions (BLF)

Non-verbal features Gaze Gaze  Contribution of the gaze to the addressee detection is dependent on: participants’ location (visible area), utterance length, current meeting action  Turn-taking behavior and addressing behavior Gesture ( pointing at a person) Gesture ( pointing at a person)  TALK_TO (X,Y) AND POINT_TO (X,Y)  TALK_TO( X,Y) AND POINT_TO (X,Z) – X talk to Y about Z

Context categories Bunt: “totality of conditions that may influence understanding and generation of communicative behavior” Bunt: “totality of conditions that may influence understanding and generation of communicative behavior”  Local context is an aspect of context that can be changed through communication Context categories: Context categories:  Interaction history ( verbal and non-verbal)  Meeting action history  Spatial context (participants’ location, distance, visible area, etc. )  User context (name, gender, roles, etc. )

Towards an automatic addressee detection Manual or automatic features annotation? Manual or automatic features annotation? An automatic target interpreter has to deal with uncertainty An automatic target interpreter has to deal with uncertainty Methods: Methods:  Rule-based method  Statistical method ( Bayesian networks)

Rule-based method 1. Processing information obtained from the utterance ( linguistic markers, vocatives, DA). The result is a list of possible addressees with corresponding probabilities 1. Eliminate cases where target is completely determined (for instance, name in vocative form) 2. Set of rules for BLF 3. Set of rules for FLF 2. Processing gaze and gesture information adding the additional probability values to the candidates

Meeting actions and addressee detection Automatic addressee detection method can be applied to the whole meeting Automatic addressee detection method can be applied to the whole meeting Knowledge about the current meeting action as well as about meeting actions history may help to better recognize the addressee of a dialogue act. Knowledge about the current meeting action as well as about meeting actions history may help to better recognize the addressee of a dialogue act.

Future works Development of multimodal annotation tool Development of multimodal annotation tool Data annotation for Data annotation for  training and evaluating statistical models  obtaining inputs for rule-based methods New meeting scenarios for research in addressing New meeting scenarios for research in addressing