On Defining Cephalic Gesture Categories Natalya V. Sukhova, Yulia V. Nikolaeva Institute of Linguistics RAS, Lomonosov Moscow State University NUST MISiS

Outline of the Talk About the Corpus. Annotation of Kinetic Phenomena. General Concept. Cephalic Gestures. Approaches to Analysis. Annotation of Cephalic Gestures. Our Approach. 9-11.06.2017 ICMC 2017, Osnabrück

1a Language As It Is: Multimodal (Multichannel) Resource Goal is to create a resource approaching the actual richness of human discourse Registered phenomena: verbal structure prosody eye gaze head movements hand gestures torso position 9-11.06.2017 ICMC 2017, Osnabrück

Kinetic / visual modality 1b Spoken Multimodal Discourse Discourse Vocal modality Verbal channel Prosody Kinetic / visual modality Eye gaze Posture Gesture Language? Other modalities Proxemics Other channels Facial expression Intonation Many other components Hands Head Most multimodal studies 9-11.06.2017 ICMC 2017, Osnabrück 4

1с RuPeCS Russian Pear Chats and Stories (Pear Film, Chafe ed. 1980): 24 conversations 96 native speakers total duration is 10 hours 110 000 words 11 media files for each conversation 24 written retellings Project web-site: www.multidiscourse.ru 9-11.06.2017 ICMC 2017, Osnabrück 5

1d The Scene Tobii glasses Listener Listener Narrator Reteller Нужно ли добавить подробности про процедуру на сам слайд? Reteller Commentator Commentator 6 6 6 6

2a Manual Gestures Communicatively significant spontaneous hand/arm movements, accompanying speech Discourse functions Adding to speech contents, facilitating verbalization, appealing to an interlocutor, regulating turn-taking, supporting rhythmical structure Standing out as a figure against kinetic background (other movements and stillness periods) Kinetic structure Distinct articulatory effort, velocity, trajectory, movement direction, hand shape and orientation, gesture range, location in gesture space 9-11.06.2017 ICMC 2017, Osnabrück

2b Distinguishing Movements and Positions Neutral (resting) positions: set of preferred hand/arm positions a speaker regularly returns to. Hand/arm posture changes: hand/arm resting position changing movements. Self-adaptors (adaptors, manipulators): physiologically motivated movements or movements with practical purposes. Gestures Typical kinetic patterns of a speaker form a gesticulation portrait (cf. ‘prosodic portraits’ in Kibrik 2009). Здесь я скажу, что у рук все очень хорошо различается, гораздо лучше, чем у головы hand/arm resting position changing movements - вот это не очень понятно, что имеется в виду? 9-11.06.2017 ICMC 2017, Osnabrück

3a Cephalic Gesture Categories MOVEMENT is all motor activity of a head. GESTURE is all motor activity that potentially has a meaning. HEAD POSTURE (point, position) brings about a change of a head position in at least one direction and may (or may not) last for some time. 9-11.06.2017 ICMC 2017, Osnabrück

3b Levels of Analysis Form – physical description Meaning – e.g. nods, shakes Function – a role in a discourse, e.g. backchannelling See Hadar et al. 1985; Bull 1987; McClave 2000; Allwood, Cerrato 2003; Benoit, Caplier 2005; Kousidis et al. 2013. 9-11.06.2017 ICMC 2017, Osnabrück

3c Our Approach to Cephalic Gesture Categories. Tiers in ELAN What does it show? Its assignment Cephalic Gesture Chain A set of cephalic gestures going together without a pause To define a structural unit of gestures (form) Cephalic Gesture Tags Comments A movement which is physiologically prominent, visible and potentially meaningful for communication To single out a salient head movement from a motor flux (form). Tags give the physical description of a movement (form). Comments provide the meaning of some movements designating nods, shakes and jerks if any. Cephalic Posture Posture change A continuum which arises after a movement changes in any way (plane, axis): posture change brings it about To see all discourse in terms of static and dynamic units (form and function) 9-11.06.2017 ICMC 2017, Osnabrück

4a Physical Form # Plane Axis Example 1 Frontal Anteroposterior Backward / Forward 2 Horizontal Vertical Up / Down 3 Sagittal Mediolateral TurnLeft / TurnRight 9-11.06.2017 ICMC 2017, Osnabrück

Vocabulary of movements 4b Formal Vocabulary Vocabulary of movements Description SIMPLE Down Downward head move Up Upward head move TurnLeft Move of the head turning on the left side Straighten Head is straightened COMPOUND Backward-forward Head moves backward and forward in a single continuous movement Slide Displacement of the head from side to side TurnRepeated Head turns from side to side (shake) DownUp (UpDown) Repeated Head moves up and down in a single continuous movement (nod) 9-11.06.2017 ICMC 2017, Osnabrück

Picture in ELAN 9-11.06.2017 ICMC 2017, Osnabrück

CONCLUSIONS & PERSPECTIVES Current annotation scheme: provides detailed annotation of movements relevant in communication; takes into account most frequent complications; makes a part of a consistent annotation system for several kinetic channels (presuming that they are used simultaneously to pass along one message); allows comparing head and manual gestures to vocal channel units and other kinetic channels’ units. Further research: extending annotated subcorpus; measuring gesticulation portrait features. 9-11.06.2017 ICMC 2017, Osnabrück

Thank you for attention! sukhova.natalya@gmail.com 9-11.06.2017 ICMC 2017, Osnabrück