1. Interaction, Media and Communication Research Group Drawing Things Together: Integrating Modalities in Dialogue. MAGIC: Multimodality and Graphics in Interactive Communication. EPSRC/ESRC PACCIT Initiative: People at the Centre of C&IT. Pat Healey, James King, Charlie Peters. Information, Media and Communication Research Group, QMUL. John Lee, Jon Oberlander. Human Communication Research Centre, University of Edinburgh. Simon Garrod, Nick Fay. Department of Psychology, University of Glasgow. Ichiro Umata, Yasuhiro Katagiri, ATR Media Integration and Communications Laboratories, Kyoto.

2. Interaction, Media and Communication Research Group Outline 1.Drawing-in-interaction: ethnographic observations turn-taking and topic management 2.Graphical Language Games: Musical Pictionary Experiment 1: Community-specific Graphical Languages Experiment 2: Mechanisms of Interaction 3.The mechanisms of interaction available to people directly constrain the form and organisation of shared symbol systems –not just individual cognitive-computational abilities 4.Augmented Human Interaction.

3. Interaction, Media and Communication Research Group Design for Human-Human Interaction

4. Interaction, Media and Communication Research Group Design for Human-Human Interaction? Single User Baton Design: Multi-User Baton Design: designed for exchange: symmetric smooth centrally balanced

5. Example: Architectural Design

6. Interaction, Media and Communication Research Group Example: Architectural Design

7. Interaction, Media and Communication Research Group Ethnography of Architectural Design 6 staff working on a design competition –mixed expertise and responsibilities Prepare 4 A2 presentation boards addressing: –use of site, environmental concerns, building use, open-space use 58 project interactions video taped over 4 weeks –40 two-party interactions –13 three-party interactions –5 interactions > 3-party One 12 minute, 3 party interaction transcribed for analysis –coded for overlaps, pauses, run-throughs, stress etc. –coded for gesture and drawing activity

8. Interaction, Media and Communication Research Group Basic Observations Complex variety of drawing spaces (in 12 minutes) –6 physical drawing spaces pieces of paper or regions of a piece –3 gestural drawing spaces drawing in the air with pen or finger –verbal spaces

9. Interaction, Media and Communication Research Group Excerpt 1

10. Interaction, Media and Communication Research Group Turn Taking Drawing activity does not automatically claim the floor: 1. Drawing activities continue across turn and speaker changes –overlapping speech is avoided (e.g., Levinson, 1983) –gestures are used to bid for and maintain floor control (Bavelas, et. al. 1995) - but see Furuyama. 2. Where competition for the floor occurs during drawing: drawing is suspended gesture & speech used to compete for floor

11. Interaction, Media and Communication Research Group Excerpt 2

12. Interaction, Media and Communication Research Group Topic Management Drawing activity is used to manage topic changes: ceases or shifts location if topic changes In excerpt 1 topic changes from rail to space. –J moves hand to draw in air above the board –reference to space is harder to resolve When topic returns to rail drawing resumes on the page (Earlier in interaction J draws same space over the drawing)

13. Interaction, Media and Communication Research Group Observations Drawing, gesture and speech are integrated into composite communicative signals (cf. Neilson and Lee, 1994) often treated as separate channels (e.g., Netmeeting) Drawing activities often serve interactional functions not only representation of domain or computational aid. use of space to manage topics cf. topographic and referential gesture spaces

14. Interaction, Media and Communication Research Group Graphical Dialogue 50% of everyday drawings are produced as part of interaction (van Sommers, 1994) Examples of Graphical Dialogue: Routine interactions –e.g., sketch maps, explanatory diagrams, games Auxiliary mode of Communication –e.g., cross-linguistic communication, aphasia Specialised interactions –e.g., design interactions Does interaction affect representation?

15. Interaction, Media and Communication Research Group Graphical Language Games: Musical Pictionary What do we do if we dont share a symbol system? Criteria for Task: communication task structured, regular, domain few (or no) established representational conventions exclusively graphical interaction Typical Set-up: Subjects seated in separate (soundproof) rooms –communication via shared whiteboard application –30 sec piano piece each Task: draw picture of target: no letters or numbers –SAME or DIFFERENT

16. Interaction, Media and Communication Research Group Room ARoom B One piano piece each: same or different? Communicate by drawing: no letters or numbers

17. Interaction, Media and Communication Research Group Music Drawing Types: 1. Abstract: –Graph-like representation of domain structure e.g., pitch, intensity, rhythm 2. Figurative: –Ad hoc associations: faces, figures, objects or situations 3. Composite: –Mixture of Abstract and Figurative (independent classification by 2 judges: Kappa = 0.9, N =287, k= 2)

18. Sequence of Figurative Trials:

19. Sequence of Abstract Trials:

20. Interaction, Media and Communication Research Group Abstract / Figurative Contrast: Abstract drawings provide a more complex representational system for the task. Specifically: 1.Systematicity: support direct comparison within and between items 2.Proto-Compositionality: distinct parts of the drawing refer to distinct parts of the music Figurative drawings are more holistic and more ad hoc.

21. Interaction, Media and Communication Research Group What affects use of Abstract or Figurative? Previous (non) findings: 1.No effect of target on drawing type: –genre / tempo / mode 2.No effect of medium on drawing type –stylus vs. mouse 3.No difference in drawing effort –same average quantity of lines and ink

22. Interaction, Media and Communication Research Group Effects of Interaction: 1.Repetition promotes abbreviation -but only if participants can signal understanding –c.f. grounding in dialogue 2.Dialogue partners tend to use drawings of the same type. –c.f. accommodation/entrainment 3. Level of communicative interaction: Concurrent Drawing –60% Abstract Alternate Drawing: –60% Figurative

23. Interaction, Media and Communication Research Group Hypotheses: 1.AUTONOMOUS CO-ORDINATION (truth wins): Participants independently migrate toward the representational scheme that is most efficient for the Concurrent task. –co-ordination of time-based axis? more comparative? –co-ordination emerges as aggregate individual experience (e.g., Clark, Lewis…) 2. COLLABORATIVE CO-ORDINATION Participants use the opportunities afforded by concurrent interaction to establish a co-ordinated 'sub-language'. –collaborative revision and refinement of conventions –co-ordination emerges through local histories of interaction

24. Interaction, Media and Communication Research Group Experiment 1: Community Sub-Languages? Does interaction contribute anything, in addition to individual expertise, to the co-ordinated use of the Abstract drawings? Experimental Design: Phase 1: develop several communities with equivalent task experience but different interaction histories Phase 2: compare interaction within and between communities. Note: community membership is hidden from participants.

25. Interaction, Media and Communication Research Group Subject 2 Subject 1Subject 6 Subject 5 Subject 3Subject 4 Round 1 = Round 2 = Round 3 = Round 4 = Phase1: Community Development

26. Interaction, Media and Communication Research Group Phase 1: 10 communities (66 people) –seated round edge of large PC lab Music Task: one piece each, same or different? –4 rounds of 12 trials –different partner on each round During Phase 1: common interaction history accumulates –reliable increase in speed: 53 sec. to 43 sec. –reliable increase in accuracy 37% to 52%

27. Interaction, Media and Communication Research Group Between Group = Within Group= Phase 2: Experimental Manipulation

28. Interaction, Media and Communication Research Group Results: Chi 2 (2) =19.0, p=0.00

29. Interaction, Media and Communication Research Group Results:

30. Interaction, Media and Communication Research Group Conclusions: Experiment 1 Cross-group interaction de-stabilises use of Abstract drawings independently of –indivdual expertise –what is being represented –individual cognitive-computational abilities –explicit knowledge of community membership (cf. Healey 1997). Support for Collaborative Co-ordination Hypothesis: localised patterns of interaction lead to community-specific (graphical) dialects Why?

31. Interaction, Media and Communication Research Group Why does interaction matter? Graphical Interaction Mechanisms: localisation alignment. Abstract drawings are proto-compositional: interaction allows participants to co-ordinate meaningful elements of each others drawings

32. Interaction, Media and Communication Research Group Hypotheses: 3. MUTUAL MODIFICATION: coordinated use of Abstract drawings depends on ability to use interaction devices to annotate and modify elements of each other's drawings. (e.g., circling, underlining, and arrows) Experiment 2: interfere with use of interaction devices (localisation and alignment) and assess effects on communication.

33. Interaction, Media and Communication Research Group Manipulation 1: Block Localisation Room A Screen Room B Screen Annotation of others drawing blocked.

34. Interaction, Media and Communication Research Group Manipulation 2: Block Alignment Room A Screen Room B Screen Transpose

35. Interaction, Media and Communication Research Group Experimental Design: Pairs seated in separate soundproof rooms Communicate via shared whiteboard- no letters or numbers Subject As View Subject Bs View + BLOCKING - BLOCKING - TRANSNPOSITON + TRANSPOSITION

36. (Blocking: Chi 2 (3) = 96.70, p =0.00, Transposition: Chi 2 (3) = 81.61, p =0.00) Effects of Interference with Interaction:

37. Interaction, Media and Communication Research Group Conclusions Experiment 2: Blocking and Transposition cause change in graphical conventions –independently of what is being represented –Independently of individual cognitive-computational abilities Complexity of emergent symbol system depends on mutual modification (localisation, alignment) –co-ordinated manipulation of external representations –participants ability to modify each others representations not editing / annotation / revision per se

38. Interaction, Media and Communication Research Group General Conclusions: Significant parallels between verbal and graphical dialogue –grounding, accommodation, turn-taking, –modality independent, interactional constraints on representation How do people co-ordinate despite differences in interpretation? Local, surface-based, manipulations of external representations. Verbal and graphical repair mechanisms –Localisation specficity, reprise fragment clarifications –Alignment sequential relevance, embedded repair

39. Interaction, Media and Communication Research Group General Conclusions: Drawing supports transactional and interactional functions not just a representational medium Not explained by physical / perceptual / computational properties per se alternate vs. concurrent drawing Modalities combine to form composite communicative signals applications often treat them as separate channels (e.g., net-meeting) Generic interaction mechanisms are apparent across modalities turn-taking, topic management, repair, grounding, accommodation Augmented interaction mechanisms richer more expressive languages?

40. Interaction, Media and Communication Research Group Augmented Human Interaction: Beyond Face-to-Face 1.Capture: gesture, expression, attention, engagement, understanding… 2.Provide augmented cues for interaction 3.Enable richer, more robust, forms of human communication –gestural languages, musical languages, dance …

41. Interaction, Media and Communication Research Group Augmented Human Interaction: Beyond Face-to-Face 1.Capture: gesture, expression, attention, engagement, understanding… 2.Provide augmented cues for interaction 3.Enable richer, more robust, forms of human communication –gestural languages, musical languages, dance …

42. Interaction, Media and Communication Research Group Augmented Human Interaction Lab multi-user, real-time, full body, motion capture. integrated 3D audio display

43. Interaction, Media and Communication Research Group Augmented Human Interaction Lab