MULTIMODAL PERSPECTIVES IN VIDEO-BASED INTERACTIONS Maria Grazia Sindoni, University of Messina

Outline  Video data  Definition of genre  Resources: mode-switching, proxemics, kinetics and gaze  Transcription and annotation

Digital video data  Multi-party room video client -> instant message programme (i.e. Camfrog)  Chat, file sharing, snapshots, watch and broadcast videos  Speakers: EML, ESL, EFL  gaze4.wmv gaze4.wmv

One digital context…

Multiple “real” contexts

What is a video chat?  It imitates face-to-face interactions  It is a form of computer mediated discourse (Herring 1996)  Shared context/s?  Delocalization  Illusory perception of a co-constructed place/setting

Medium affordances  In skype there is no need to regulate turn-taking. It is more similar to face-to-face interactions  In Camfrog or other multi-party client systems:  Spoken Floor -> only one speaker at a time  Written Floor -> WTs are linearized as they are published in chronological order  gaze2.wmv gaze2.wmv

Mode-switching (MS): speech and writing  It paraphrases the linguistic notion of “codeswitching” (Heller, 1988)  Halliday: mode as the communicative channel (i.e. oral vs. written discourse)  MS is the alternation between speech and writing in the same communicative event  When? Why?

MS and conversation analysis  MS can be self-initiated or other-initiated  Aims: repair trouble in conversation; address technical problems; hold parallel floors; other communicative purposes: tell a secret, give specific information (numbers, address); etc.  Refs. from conversation analysis: Sacks, Schegloff,, Jefferson 1974; Sacks 1992; etc.

Interviews: How often do you videochat?

Reasons why informants use written comments during a videochat

SecrecyDo not want to be heard/ tell a secret Intimacy Communicate particularly intimate feelings (e.g. love, friendship, loyalty etc.) or express face threatening feelings (embarrassment, shame, taboos, etc.) Fun/Kidding Add visual (e.g. through the use of emoticons) or expressive (e.g. through swear words) power to communication Preciseness Give precise information (e.g. addresses, telephone numbers, names, references, etc.) OtherNot well aware about when and why they resort to written comments

Example 1: other-initiated MS  1. Essence: do you see it? How it’s different/ like you were on the (…) version is (…) /is on my mic is like going up// like in a circle/ or whatever/ that’s what I was asking if it was too loud or not//  2. Diamond: ah ok  3. Diamond: who here is married?  4. Essence: Tinker [mode-switch]  5. Essence: no/it’s not too loud/because it’s not on the red// [mode-switch]

Example 2: self-initiated MS  1. Tod3344: ARE MY FONTS RIGHT NOW?  2. Essence: yes tod/ tod you can make it small/ you know/ make it bigger/ like on the 16// 14 something like that/ used for it like this/ y’know//  3. Diamond: Tod3344, Please Respect all users.  4. Essence: they<small… but u changed on red [mode-switch] 

Example 3: self-initiated MS for repair  1. Tinker: Sensei your video’s fine  2. Tinker: no I mean/ Sensei your audio’s fine// [mode-switch]

Proxemics: social distance  Perception, use, structuring and management of space  Social distance in videochat is fixed and medium constained  Arrangement of space in interaction (sociopetal vs. sociofugal, Hall 1966)  Erosion of traditional proxemic patterns

Proxemics: perception, use, structuring and management of space Social distance in videochat is fixed and medium constrained

Proxemics

Kinesics: movements and posture  Variations across cultures, societies, individuals (Kendon 1990, 2004)  Body movements and intentionality (Norris 2004)  intentional: deictic gestures  halfway: iconic or metaphoric gestures  unintentional: posture  Frozen/living image

Variety of actions/movements…

Kinesics: movements and posture

Kinesic sequence

Eating…

Drinking…

Smoking…

Calling over the phone…

Yawning…intentional?

Get ready for the night…

Absence

Gaze and eye contact  Turn-taking and gaze: Goodwin (1980, 1981) and Kendon (1967, 1990)  Problems in turn-taking: telephone, mediated conversation, blind/sighted interaction (Everts 2004)  Gaze in video chat: impossibility of eye contact, impossibility to reciprocate gaze

Gaze: impossibility of eye contact, non reciprocal gaze

Impossibility of eye contact

Self-looking

Transcription and annotation  Issues in video analysis transcription and annotation (Thibault 2000; van Leeuwen, Jewitt 2001; Norris 2004; Pink 2007; Flewitt et al. 2009)  A model entirely based on writing fails to reproduce seminal aspects (Sindoni 2010; 2011; 2012; 2013; 2014).  A multimodal model needs visual resources, such as screenshots and/or drawing.

Guidelines ..\MoM\guidelines_Videochat.docx..\MoM\guidelines_Videochat.docx

Selected references  Azzarello, E. (website).  Flewitt, R. et al. (2009). What are multimodal data and transcription? In C. Jewitt (ed.). The Handbook of Multimodal Analysis. London & New York: Routledge.  Norris, S. (2004). Analyzing Multimodal Interaction. A Methodological Framework. London and New York: Routledge.  Pink, S. (2007) [2001]. Doing Visual Ethnography. London: Sage.  Sacks, Harvey, Emanuel A. Schegloff & Gail Jefferson A simplest systematics for the organization of turn-taking for conversation. Language. 50,  Sacks, Harvey Lectures on Conversation. Oxford : Blackwell.  Sindoni, M. G. (2010). Models of verbal and non-verbal interaction in Web 2.0 textuality. Bérénice. XVI(43): pp  Sindoni, M. G. (2011b). Online conversations. A sociolinguistic investigation into young adults’ use of videochats. Classroom Discourse. II(2): pp  Sindoni, M. G. (2013). Spoken and Written Discourse in Online Interactions. A Multimodal Approach. London & New York: Routledge.  Thibault, P. J. (2000). Multimodal transcription of a television advertisement: theory and practice. In A. P. Baldry (ed.). Multimodality and Multimediality in the Distance Learning Age Campobasso: Palladino.  van Leeuwen, T., Jewitt, C. (eds.). (2001). The Handbook of Visual Analysis. London: Sage.