Details Regarding Stimuli

Slides:



Advertisements
Similar presentations
Gesture and speech are controlled by the same system Maurizio Gentilucci Dipartimento di Neuroscienze, Università di Parma, Italy RTM (Rete Multidisciplinare.
Advertisements

National Accessible Reading Assessment Projects Formulating Public Opinion on Definitions of Reading Proficiency Christopher Johnstone Council for Exceptional.
18 and 24-month-olds use syntactic knowledge of functional categories for determining meaning and reference Yarden Kedar Marianella Casasola Barbara Lust.
Using prosody to avoid ambiguity: Effects of speaker awareness and referential context Snedeker and Trueswell (2003) Psych 526 Eun-Kyung Lee.
Culture, Communication Practices, and Cognition: Selective Attention to Content Versus Context Keiko Ishii Hokkaido University, Japan.
Evaluating the Effects of Text Chunking on Subtitling: A Quantitative and Qualitative Examination Andrew T. Duchowski COMPUTER SCIENCE CLEMSON UNIVERSITY.
Why do bilinguals gesture more than monolinguals? Elena Nicoladis University of Alberta.
Frames of Reference for Perception and Action in the Human Visual System MELVYN A. GOODALE* AND ANGELA HAFFENDEN Department of Psychology, University of.
Evaluating the Validity of NLSC Self-Assessment Scores Charles W. Stansfield Jing Gao Bill Rivers.
Learning About Autism Clip 1 – How do you feel about being autistic? Clip 2 – Do you like being autistic?
The fragile-X syndrome: What about the deficit in the pragmatic component of language? Abstract The fragile-X syndrome: What about the deficit in the pragmatic.
Common Ground Linguistic referents are established w/in a “domain of interpretation”, which includes context –One component of context = Common Ground.
Comprehension of Grammatical and Emotional Prosody is Impaired in Alzheimer’s Disease Vanessa Taler, Shari Baum, Howard Chertkow, Daniel Saumier and Reported.
Administering ELDA K & ELDA 1-2 English Language Development Assessment Assessing ELL Students in the Primary Grades Developed by the Limited English Proficient.
Asperger Syndrome Learning to Cope. What is Apserger Syndrome Asperger syndrome (AS) is a developmental disorder that is characterized by: 1 limited interests.
Frank E. Musiek, Ph.D., Jennifer Shinn, M.S., and Christine Hare, M. A.
Suppression Task. We used a task modeled on Gernsbacher et. al. (1991) Lewis and colleagues 3,4 measured the monocular visual field extent of 3-, 4-, and.
Reicher (1969): Word Superiority Effect Dr. Timothy Bender Psychology Department Missouri State University Springfield, MO
Electrophysiological Processing of Single Words in Toddlers and School-Age Children with Autism Spectrum Disorder Sharon Coffey-Corina 1, Denise Padden.
The effects of working memory load on negative priming in an N-back task Ewald Neumann Brain-Inspired Cognitive Systems (BICS) July, 2010.
Tonal Violations Interact with Lexical Processing: Evidence from Cross-modal Priming Meagan E. Curtis 1 and Jamshed J. Bharucha 2 1 Dept. of Psych. & Brain.
Effect of laterality-specific training on visual learning Jenna Kelly & Nestor Matthews Department of Psychology, Denison University, Granville OH
WP6 Emotion in Interaction Embodied Conversational Agents WP6 core task: describe an interactive ECA system with capabilities beyond those of present day.
Sperry Physiological Psychology The Core Studies.
Baron-Cohen Cognitive Psychology The Core Studies.
Poster presented at APS 2014 Abstract This study was conducted to determine if explaining criminal behavior influences later identification. Schooler and.
Simultaneous interpreting is a particularly demanding cognitive process in which language comprehension and production take place at the same time in two.
Gaze cues in mother-child dyads Heather Bell and Meredith Meyer University of Oregon INTRODUCTION RESULTS CONCLUSIONS METHODS REFERENCES ACKNOWLEDGEMENTS.
Presentation by Peggy Yost.  Aspergers Syndrome (AS) belongs to a group of childhood disorders known as pervasive developmental disorders or autistic.
Telepractice in the University Speech and Language Clinic:
Differences in the Semantic Relatedness of Signs vs
‘Signed and Spoken Thinking‘?
Emotion Knowledge in Maltreated Preschoolers
ADHD, ADD and Autism Supported by.
Gallaudet University INT 492, Spring 2017
Autism Spectrum Disorder
Ten Myths of Multimodal Interaction
1.WHAT IS AUTISM? Autism Spectrum Disorder, ASD, and autism are both general terms for a group of complex disorders of brain development. These disorders.
Communication Leadership Skill Area
Academic Writing Skills
Application of Our Problem: Biological Correlates of Language
LANGUAGE (Speech/Language Impaired)
Investigating the combined effects of word frequency and contextual predictability on eye movements during reading Christopher J. Hand Glasgow Language.
Assist. Prof. Dr. Ilmiye Seçer Fall
A Child with Autism Can Swim
Autism and Asperger Syndrome
Communication Skills “Communication” is the process by which ideas, feelings, and information are shared. It involves the skills of listening, speaking,
Bi-dialectalism: the investigation of the cognitive advantage and non-native dialect perception in noise Brittany Moore, Jackie Rayyan, & Lynn Gilbertson,
Phonological Priming and Lexical Access in Spoken Word Recognition
Approaches to Discourse Analysis
Sarah.
Project title All Wales Annual Equality Briefing Hidden Disabilities
RESULTS AND DISCUSSION
Susan Geffen, Suzanne Curtin and Susan Graham
Aylin Küntay Meeting 6 Language and Communication Disorders
Susan Geffen, Suzanne Curtin and Susan Graham
Autism Spectrum Disorder
The involvement of visual and verbal representations in a quantitative and a qualitative visual change detection task. Laura Jenkins, and Dr Colin Hamilton.
Business Communication
Business Communication
Chapter 29 Emerging Technologies and the Generation of Knowledge
Chapter 4 – Communication Skills
Detection of Human Simultaneity Thresholds Using Cross-Modal Stimuli
The Nature of Learner Language (Chapter 2 Rod Ellis, 1997) Page 15
An Introduction to Speechreading
Communication Leo Africano
Partners in Learning Educator Professional Development
Child Outcome Summary Form
Unit 3 Biological Bases of Behavior
Perceptual-Motor Deficits in Children with down syndrome: Implications for Intervention Study by: Naznin Virji-Babul, Kimberly Kerns, Eric Zhou, Asha.
Presentation transcript:

Details Regarding Stimuli Integration of Gesture and Speech in Real-Time Reference Resolution Ellen Campana†, Laura Silverman‡, Michael K. Tanenhaus†, and Loisa Bennetto‡ †Department of Brain and Cognitive Sciences University of Rochester ‡ Department of Clinical and Social Sciences in Psychology University of Rochester Non-deictic manual gestures are ubiquitous in language production. People produce gesture even when their interlocuters cannot see them [1] and can experience difficulty speaking when their hands are restricted [5]. There is some consensus that the production system allows different aspects of a single message to be expressed simultaneously over, and distributed between, linguistic utterances and gestures [2]. However, researchers disagree about the role of gesture in comprehension; whether it is ignored [6], processed separately [4], used only when speakers are having difficulty [8], or immediately integrated with the content of the co-occurring speech [7]. The present experiment provides evidence in support of immediate integration between the two modalities. Overview In the example shown in the method section the actor in the center of the screen describes the contents of the top left-hand quadrant, saying “a mitten and a curved line with one loop” as she gestures the shape of the loop. In the speech-only version of this trial she says “a mitten and a curved line with one loop” without moving her hands. . Details Regarding Stimuli Note that in this example the point of disambiguation (POD) with respect to the speech alone occurs just after the start of the word “one.” . Conclusions & Future Work The results from the present experiment demonstrate that listeners immediately integrate information from speech and co-occurring gesture to create a meaningful message as an utterance unfolds. In particular, the when descriptive gestures are present, listeners attend to relevant gestures and appear to process the speaker’s meaning even before it is fully articulated via speech. Our future work involves extending the present methodology to examine gesture and speech comprehension in individuals with high-functioning autism and Asperger Syndrome. Individuals with autism have significant social and communicative impairments and specifically have difficulties interpreting the non-verbal behaviors of others, such as gaze and facial expressions. By examining speech and gesture integration, we can learn the extent to which difficulties with gesture comprehension may contribute to their overall social and communicative difficulties. Furthermore, studying the timing of speech and gesture integration may also illuminate whether individuals with autism have an underlying impairment in processing information across modalities, which may affect their functioning in other realms. The findings from this research have important treatment implications, and may also shed light on the role of multimodal integration in typical populations. Method Ten participants watched 14 videos of a woman describing simple shapes on a display in which the video was surrounded by four potential referents: the target, a speech competitor, a gesture competitor, and an unrelated foil. The task was to “click on the shape that the speaker was describing”. . In half of the 14 videos the speaker used a natural combination of speech and gesture [3]. In the other half, the speaker’s hands remained in her lap. The stimuli were counterbalanced in two lists such that across participants each set of visual stimuli occurred with both speech+gesture videos and speech-only videos. The speech-only videos were directly modeled on the natural speech+gesture videos and contained an identical verbal description. . [1] Alibali, M. W., Heath, D. C., and Myers, H. J. (2001). Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen. Journal of Memory and Language, 44, 1-20. [2] Alibali, M. W., Kita, S., and Young, A. J. (2000). Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes, 15, 593-613. [3] Campana, E., Silverman, L., Tanenhaus, M. K., and Bennetto, L. (2004). Iconic gesture production in controlled referential domains. In Proceedings of the 26th Annual Meeting of the Cognitive Science Society. Chicago, IL. [4] Goldin-Meadow, S., and Singer, M. A. (2003). From children’s hands to adults’ ears: Gesture’s role in the learning process. Developmental Psychology, 39, 509-520. [5] Graham, J. A., and Heywood, S. (1975). The effects of elimination of hand gestures and of codability on speech performance. European Journal of Social Psychology, 2, 189-195. [6] Krauss, R. M., Dushay, R. A., Chen, Y., and Rausher, F. (1995). The communicative value of conversational hand gestures. Journal of Experimental Social Psychology, 31, 533-553. [7] McNeill, D., Cassell, J., and McCullough, K.-E. (1994). Communicative effects of speech-mismatched gestures. Research on Language and Social Interaction, 27, 223-237. [8] Rauscher, F. H., Krauss, R. M. and Chen Y. (1996). Gesture, speech, and lexical access: The role of lexical movements on speech production. Psychological Science, 7, 226-231.. . References Results Speech+Gesture 0.1 0.2 0.3 0.4 0.5 0.6 S+G Target S+G Speech Competitor S+G Foil 1 2 3 Participants are more likely to click on the correct target before reaching the POD in the gesture+speech condition, suggesting that gesture information speeds processing. Participants are more likely to look at the speech competitor in the speech-only condition, suggesting that gesture information reduces competition. Eye-movement graphs reveal that participants seem to experience longer-lasting competition from the speech competitor in the speech-only condition. . Speech-only 0 ms POD (2112 ms) 3000 ms Speech+Gesture We gratefully acknowledge the hard work of the undergraduate members our research group: Stephanie Packard, Hsi-Mei (Betty) Huang, Kathryn McNamara, Kelley Knock, and Sandra Vanegas. 0 ms POD (2178 ms) 3000 ms