Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh.

Slides:

Advertisements

Similar presentations

APPROACHES TO T&L Language

Advertisements

Towards Human-Realistic Animation by Observing Real Human Dynamics Darren Cosker Towards Human-Realistic Animation by Observing Real Human Dynamics Darren.

Encouraging enterprise Moving towards a zero-waste society Developing a capable population Fostering resilient communities Advancing global citizenship.

Probabilistic Tracking and Recognition of Non-rigid Hand Motion

January 10, Participants will gain an awareness of:  Timeline for Professional Development and Implementation  Relationship Between the Common.

Phonetics as a scientific study of speech

1 © 2005 CHIL KTH ASIDE 2005, Aalborg, Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson Applications.

Mothers and Children in Conversation: Cultural & Gender Variations Dana Brown Department of Applied Psychology New York University Special thanks to Dr.

Emmanuel A Stamatakis Centre for Speech, Language and the Brain, Department of Experimental Psychology, University of Cambridge School of Psychological.

Weekly Nugget #3 October 13, 2011 Tips for Rapport Building Amanda Kilgore - MI Coordinator.

Delivering Your Speech. Why Is Delivery Important? Delivery: The way you communicate messages orally and visually through your use of voice, face, and.

Empirical and Data-Driven Models of Multimodality Advanced Methods for Multimodal Communication Computational Models of Multimodality Adequate.

Collection and Analysis of Multimodal Interaction in Direction Giving Dialogues Seikei University Takeo TsukamotoYumi Muroya Masashi Okamoto Yukiko Nakano.

Providing Artifact Awareness to a Distributed Group through Screen Sharing Kimberly Tee, University of Calgary Saul Greenberg, University of Calgary Carl.

The Use of Eye Tracking Technology in the Evaluation of e-Learning: A Feasibility Study Dr Peter Eachus University of Salford.

Maine Department of Education Maine Reading First Course Session #3 Oral Language Development.

Chapter 3 Nonverbal Communication. What is nonverbal communication? “Everything that communicates a message but does not use words” Facial expressions,

Emotion in Meetings: Hot Spots and Laughter. Corpus used ICSI Meeting Corpus – 75 unscripted, naturally occurring meetings on scientific topics – 71 hours.

Presented By: Karan Parikh Towards the Automated Social Analysis of Situated Speech Data Watt, Chaudhary, Bilmes, Kitts CS546 Intelligent.

Dan’s Flirtations with Discourse (and Narrative) ---in his approach to language acquisition “Flirting with Discourse” Language Acquisition Task + Turn.

Psycholinguistics 09 Conversational Interaction. Conversation is a complex process of language use and a special form of social interaction with its own.

Wilson, “The case for sensorimotor coding in working memory” Wilson’s thesis: Items held in short-term verbal memory are encoded in an “articulatory” format.

Discussion Silvia Lindtner INF 132 April 07. Fitts’ law - recap A predictive model of time to point at an object Help decide the location and size of.

Module 4: Building Community Leadership Capacity Leadership and Effective Communication Unit II: Effective Communication.

Building the Design Studio of the Future Aaron Adler Jacob Eisenstein Michael Oltmans Lisa Guttentag Randall Davis October 23, 2004.

Part 2: Process and Models.  The Linear View Sender encodes message Sender encodes ideas or feelings into a message channelreceiver Message is injected.

The Art of Public Speaking

PSY 369: Psycholinguistics Language Production & Comprehension: Conversation & Dialog.

Understanding Effective Communication Techniques

Human-Computer Interaction IS 588 Spring 2007 Week 4 Dr. Dania Bilal Dr. Lorraine Normore.

Giving Effective Presentations 101 (H072) Rob Hornberger Coordinator of Web and Data Support.

Data collection and experimentation. Why should we talk about data collection? It is a central part of most, if not all, aspects of current speech technology.

Chapter 13 Delivery. What is Good Delivery?  Does not call attention to itself  Blend of formality and conversation  Art, not science.

Common Ground Linguistic referents are established w/in a “domain of interpretation”, which includes context –One component of context = Common Ground.

Recognition of meeting actions using information obtained from different modalities Natasa Jovanovic TKI University of Twente.

Teaching Productive Skills Which ones are they? Writing… and… Speaking They have similarities and Differences.

Incorporating Extra-linguistic Information into Reference Resolution in Collaborative Task Dialogue Ryu Iida Shumpei Kobayashi Takenobu Tokunaga Tokyo.

The effects of relevance of on-screen information on gaze behaviour and communication in 3-party groups Emma L Clayes University of Glasgow Supervisor:

Multi-modal Reference Resolution in Situated Dialogue by Integrating Linguistic and Extra-Linguistic Clues Ryu Iida Masaaki Yasuhara Takenobu Tokunaga.

Collaborative Learning. (c) Frey & Fisher, 2008 TEACHER RESPONSIBILITY STUDENT RESPONSIBILITY Focused Instruction Guided Instruction “ I do it ” “ We.

Language, health and aging: notes I Course Description: Should we use or avoid a simplified speech register (Elderspeak) when speaking to older people?

Chapter 20 Choose and Rehearse a Method of Delivery and

Turn-taking Discourse and Dialogue CS 359 November 6, 2001.

ENTERFACE 08 Project 1 “MultiParty Communication with a Tour Guide ECA” Mid-term presentation August 19th, 2008.

Teaching Speaking Skills

The Effect of Interface on Social Action in Online Virtual Worlds Anthony Steed Department of Computer Science University College London.

Developed by Counseling & Wellness Services for the Department of Housing & Residential Education.

Summary of results. Reiterate goal of investigation: How general is anticipatory behavior observed by Land & McCleod? Found: Untrained subjects exhibit.

Speaking while monitoring addressees for understanding Torsten Jachmann Herbert H. Clark and Meredyth A. Krych Seminar „Gaze as function of.

Introduction: How do you keep in touch with friends and family? How do you express yourself and your feelings and organise your life? If you think about.

Public Speaking: The Communication Model. Objectives: Define Communication List and explain the components of the communication process.

1/17/20161 Emotion in Meetings: Business and Personal Julia Hirschberg CS 4995/6998.

Copyright © Allyn & Bacon 2008 The Importance of Effective Listening.

Presentations. preparing a speech find your purpose (entertain, inform, persuade) formulate your basic position on the subject (the central idea of your.

Overview People are engaged in on-going work of constructing shared understanding of the task Understanding of one another Group identity, behavior, norms.

ENTERFACE’08 Multimodal Communication with Robots and Virtual Agents mid-term presentation.

Exhibition#3 Becoming A Better me By:Luis, Saki. Non Verbal skills -Gestures,voice, and how we stand sends strong messages -When interacting we continuously.

Elements of Communication How do you communicate with your friends, family, teachers, and co-workers?

WP6 Emotion in Interaction Embodied Conversational Agents WP6 core task: describe an interactive ECA system with capabilities beyond those of present day.

Human Joint Transportation in a Multi-User Virtual Environment Stephan Streuber Astros.

COMMUNICATION MEDIA, AND . PRESENTERS: CHOGO,M,D (092SIS10). :AKPADE, YAYRA EDNA (092SIS1).

Blink Is Not A Random Event In Reading Yu-Chi Tai, James Sheedy, & John Hayes Pacific University, College of Optometry.

Details Regarding Stimuli

Literacy Content Specialist, CDE

A Dedicated Binding Mechanism for the Visual Control of Movement

Machine Learning for Visual Scene Classification with EEG Data

Fast Sequences of Non-spatial State Representations in Humans

Volume 26, Issue 9, Pages (May 2016)

Collaborative Learning

Social Sciences Librarian University of Nebraska at Omaha

Presentation transcript:

Alignment in multimodal dialogue corpora Robin Hill and Ellen Gurman Bard Edinburgh

Joint Construction Task Collaborative and cooperative joint action in a dyad. Two eye-trackers linked in parallel. Manipulate communication/feedback channels available from trial to trial: –Speech permitted/denied –Partners mouse cursor visible/invisible –Partners gaze position visible/invisible

JEL Lab

JCT in action

Cross-recurrence Distribution of visual alignment over time Contrast with random gaze alignment. Richardson, D. C., & Dale, R. (2005). Looking to understand: The coupling between speakers' and listeners' eye movements and its relationship to discourse comprehension. Cognitive Science, 29(6), Richardson, D. C., Dale, R., & Kirkham, N. Z. (2007). The art of conversation is coordination: Common ground and the coupling of eye movements during dialogue. Psychological Science, 18(5),

Cross-recurrent Gaze Richardson & Dale, 2005

In the JCT corpus Eye-movement records for the central 120 seconds of each trial used. Time-linked overlap = percentage of time with JA gaze-tracks in same Region of Interest for a series of 200ms time lags across a moving 12-second temporal window. Baseline = randomly re-ordered series of the same eye- movement record. If the two are identical, temporal overlap purely by chance.

Global mean recurrence Gaze at dynamic objects well coordinated Greatest overlap simultaneous Strongest coincidence approximately in the range ±2000ms

Global mean recurrence Symmetrical distribution indicates similar behaviour exhibited by both partners. Similar to the R&D findings

Joint attention and communication modality Coordination is non-random in all conditions including baseline: Joint task always results in some level of gaze coordination.

Speech Mouse Gaze Mouse Gaze Speech Gaze Gaze Speech Mouse Mouse SpeechNone

Speech summary Speech appears to reduce the occurrence of visual alignment. Speech tends to increase task completion times. BUT People make fewer mistakes during the task when they can speak. Speech-improved strategies persist even if dyads are later prevented from talking to each other.

Next… Examine cross-recurrence pinned to specific linguistic elements/events. –E.g. referring expressions.

Eye-voice-eye Synchronising of gaze can be broken into two components: Eye-voice span (speaker) + Voice-eye span (speaker-listener) Examine the relationship to determine if the communicative burden is slanted towards the speaker of the listener. –E.g. Are there longer production times and do these lead to shorter comprehension times?

Thank you.