1. Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh

2. Joint Construction Task Collaborative and cooperative joint action in a dyad. Two eye-trackers linked in parallel. Manipulate communication/feedback channels available from trial to trial: –Speech permitted/denied –Partners mouse cursor visible/invisible –Partners gaze position visible/invisible

3. JEL Lab

4. JCT in action

5. Cross-recurrence Distribution of visual alignment over time Contrast with random gaze alignment. Richardson, D. C., & Dale, R. (2005). Looking to understand: The coupling between speakers' and listeners' eye movements and its relationship to discourse comprehension. Cognitive Science, 29(6), 1045-1060. Richardson, D. C., Dale, R., & Kirkham, N. Z. (2007). The art of conversation is coordination: Common ground and the coupling of eye movements during dialogue. Psychological Science, 18(5), 407-413.

6. Cross-recurrent Gaze Richardson & Dale, 2005

7. In the JCT corpus Eye-movement records for the central 120 seconds of each trial used. Time-linked overlap = percentage of time with JA gaze-tracks in same Region of Interest for a series of 200ms time lags across a moving 12-second temporal window. Baseline = randomly re-ordered series of the same eye- movement record. If the two are identical, temporal overlap purely by chance.

8. Global mean recurrence Gaze at dynamic objects well coordinated Greatest overlap simultaneous Strongest coincidence approximately in the range ±2000ms

9. Global mean recurrence Symmetrical distribution indicates similar behaviour exhibited by both partners. Similar to the R&D findings

10. Joint attention and communication modality Coordination is non-random in all conditions including baseline: Joint task always results in some level of gaze coordination.

11. Speech Mouse Gaze Mouse Gaze Speech Gaze Gaze Speech Mouse Mouse SpeechNone

12. Speech summary Speech appears to reduce the occurrence of visual alignment. Speech tends to increase task completion times. BUT People make fewer mistakes during the task when they can speak. Speech-improved strategies persist even if dyads are later prevented from talking to each other.

13. Next… Examine cross-recurrence pinned to specific linguistic elements/events. –E.g. referring expressions.

14. Eye-voice-eye Synchronising of gaze can be broken into two components: Eye-voice span (speaker) + Voice-eye span (speaker-listener) Examine the relationship to determine if the communicative burden is slanted towards the speaker of the listener. –E.g. Are there longer production times and do these lead to shorter comprehension times?

15. Thank you.