To Beat or Not To Beat? Beat Gestures in Direction Giving Chris Brandhorst & Mariët Theune University of Twente

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Overview Beats and other gesture types Research context: The Virtual Guide A small direction giving corpus How to recognize beats? The Beat Filter When are beats used? Concept categories A (very) simple beat usage model Conclusions and future work

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Beats and other gestures Gesture types Deictic: pointing at an object’s location Iconic: representing the shape of a concrete object Metaphoric: depicting an abstract object using metaphor Beat: indicating discourse structure; emphasis McNeill (1992) “Hand and Mind” p.93: beats made up 44.7% of used gestures in a cartoon narration corpus.

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Context: The Virtual Guide An embodied direction giving agent in a 3D environment

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Video (link naar filmpje hier) Try out the Virtual Guide “live” here:

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Gesture generation Keyword-based Turns (“left”, “right”, etc.): fixed pointing gesture in turn direction Objects (“the coffee counter”, etc.): –pointing gesture to absolute 3D object location (gesture is computed dynamically) –pointing gesture to relative object location, from viewpoint along the route (fixed gesture; like Turns) –iconic gesture reflecting object shape (fixed gesture from “gestionary”)

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop When to use beat gestures? In the BEAT system (Cassell et al., 2001) gestures are used to mark new information and to contrast items Beats are given low priority: they are only used when no other gesture type is available In the Virtual Guide, in almost all cases a pointing or iconic gesture is available – so, no “need” for beats? No: human direction givers do often use beat gestures, as shown in our small video corpus.

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Direction giving video corpus Used for this study: 15 short video clips (± 45 sec. each) 4 different Dutch speakers, 3-4 clips each 2 different destinations in our building 2 versions of each clip (except one): with a listener present or to the camera 133 gestures; 124 annotated (others not clearly visible)

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop How to recognize beats? Beat characteristics: “A simple flick of the hand” Short and quick Only 2 gesture phases: preparation and retraction (no stroke) No “tensed stasis” Formless hand shape Formal coding, based on shape only: the Beat Filter (McNeill, 1992) “filters out” beats from other gestures.

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop The Beat Filter 1.Does the gesture have other than 2 movement phases? 2.How often does tensed stasis or finger movement appear? 3.If the first movement is in non-center space, is any other movement in center space? 4.If there are exactly 2 movement phases, are they in different spaces? Add 1 point for each “yes” answer to Questions 1, 3, 4 to the number given in answer to Question 2. The lower the score, the more likely the gesture is a beat.

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Using the Beat Filter (1) 109 gestures were scored with the Beat Filter* 95 of those were annotated by two annotators (the other 14 were used as test items) Annotator agreement on the Beat Filter questions was very low: –Question 1: K = 0.43 –Question 2: K = 0.31 –Question 3: K = 0.18 –Question 4: answer is dependent on Q1, so computing reliability makes no sense *15 gestures were considered to be “obvious” (other gesture types than beats) and not “filtered” by annotator A …!

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Using the Beat Filter (2) Agreement on total Beat Filter scores: 44.2% same score (but possibly on different grounds!) 36.8% difference of % difference of 2 2.1% difference of 3 In the end, only the scores of annotator A were used.

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Annotating gesture types Gesture types based on global shape information: resemblance to mentioned object, finger pointing, directional component, etc. (in combination with speech) Annotator agreement: Agreed on 83.3% of gesture types (102 of 124), K=0.73 –Of these, 33.3% are beats (34 of 102) Disagreed on 17.7% of gesture types (22 of 124) –Most confused were point and iconic (45.5%) –Next most confused were beat and point (13.6%)

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Gesture types and beat score Beats do have lower Beat Filter scores Many pointing gestures have low scores too *NF (Not Filtered) gesture types were not entirely obvious after all…! NF*#% beat ‘multi-beat’443.2 iconic point not agreed #

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop When are beat gestures used? Some direction giving concept categories were defined: Directions (up, down, left, right, …) (Other) Spatial information (through, in, on, at, across, …) Duration & Timing (all the way, continue, immediately, …) Landmarks –Nouns (windows, a square, the hallway, …) –Pronouns (that, the same, this, they, it, …) Points in Time or Space (now, then, here, there, …) Hesitations (uh, uhm, I would say, something like that, maybe, …)

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Concepts / gestures overview Concept#Beat (%)Other (%)Total (%) Spatial info456 (13.3%)3 (6.7%)9 (20.0%) Hesitations959 (9.5%)4 (4.2%)13 (13.7%) Duration & timing373 (8.1%)1 (2.7%)4 (10.8%) Landmarks1767 (4.0%)30 (17.0%)37 (21.0%) pronoun493 (6.1%)6 (12.2%)9 (18.3%) noun1274 (3.1%)25 (19.7%)29 (22.8%) Points in time/space1024 (3.9%)12 (11.8%)16 (15.7%) Directions841 (1.2%)39 (46.4%)40 (47.6%) Total53930 (5.6%)89 (16.5%)119 (22.1%) Othern/a4 (n/a)10 (n/a)14 (n/a)

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Concept Categories Relative frequency of concept categories: Landmarks are most frequently mentioned Directions are only in fourth place

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Concepts and gestures Not fitting into these categories: 4 beats, 10 “other gestures” SI = Spatial Information; H = Hesitations; DT = Duration & Timing; L = Landmarks; PTS = Points in Time or Space; D = Directions

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Landmarks: pronoun or noun Landmarks as pronouns: fewer gestures, relatively more beats

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop A simple beat usage model The probability that a beat gesture B is generated to accompany an utterance u (and modelling speaker s): P(B|u) = P(B|C u ) x m s where C u is the concept category of u P(B|C u ) is the probability of B accompanying C u based on corpus data m s is an optional multiplier for speaker s (weight factor)

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Toward a better model Other factors than just corpus frequency should be taken into account. For example: First or second time the same directions are given? Listener present or not? Context: influence of preceding and following concepts / gestures Etc. And of course, more (and more reliable!) corpus data are needed.

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Conclusions How can we recognize beats? Applying Beat Filter to recognize beat gestures may not give reliable results “Impressionistic” gesture type annotation was more reliable Add “directionality” and “hand shape” to Beat Filter? When are beats used? “Other” gestures don’t always take precedence over beats Beats mark spatial information, hesitations, duration and timing more often than other gestures do

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop Future work More data More reliable annotation Investigate when beats or other gestures are used given a concept category Better / more general concept categories? → Implement in the Virtual Guide

Brandhorst & Theune – Beat Gestures in Direction Giving – Gesture Workshop The End Questions?