The role of auditory-visual integration in object recognition Clara Suied 1, Nicolas Bonneel 2 and Isabelle Viaud-Delmon 1 1 CNRS – UPMC UMR 7593 Hôpital.

Slides:

Advertisements

Similar presentations

Memory and Cognition Lecture 10: Filling in the gaps… ENCODING RETRIEVAL Seeing Word Hearing Word MTL.

Advertisements

Progressive Perceptual Audio Rendering of Complex Scenes Thomas Moeck - Nicolas Bonneel - Nicolas Tsingos - George Drettakis - Isabelle Viaud-Delmon -

Analyzing Visualization Workload through Leverage Points Mark A. Livingston†, Kristen K. Liggett‡, Paul R. Havig‡, Jason A. Moore‡, Jonathan W. Decker†,

Attention and neglect.

Visual speech speeds up the neural processing of auditory speech van Wassenhove, V., Grant, K. W., & Poeppel, D. (2005) Proceedings of the National Academy.

Visual Attention Attention is the ability to select objects of interest from the surrounding environment A reliable measure of attention is eye movement.

Human (ERP and imaging) and monkey (cell recording) data together 1. Modality specific extrastriate cortex is modulated by attention (V4, IT, MT). 2. V1.

LOGO The role of attentional breadth in perceptual change detection Professor: Liu Student: Ruby.

August 16, 1999SMPC Conference '991 Cross-modal integration: Alignment of auditory and visual accent structures in motion pictures Dr. Scott D. Lipscomb.

NEUR 3680 Midterm II Review Megan Metzler

Read this article for next week: A Neural Basis for Visual Search in Inferior Temporal Cortex Leonardo Chelazzi et al. (1993) Nature.

Charles Spence Department of Experimental Psychology, Oxford University New Perspectives from the Human Sciences Consumer Focus Workshop (November, 2001)

Attention I Attention Wolfe et al Ch 7. Dana said that most vision is agenda-driven. He introduced the slide where the people attended to the many weird.

Cross-modal Prediction in Speech Perception Carolina Sánchez, Agnès Alsius, James T. Enns & Salvador Soto-Faraco Multisensory Research Group Universitat.

Saccadic eye movements cause compression of time as well as space Concetta Morrone, Ross & Burr.

Chapter 6: Visual Attention. Scanning a Scene Visual scanning – looking from place to place –Fixation –Saccadic eye movement Overt attention involves.

Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia.

Read this article for Wednesday: A Neural Basis for Visual Search in Inferior Temporal Cortex Leonardo Chelazzi et al. (1993) Nature.

Test Oct. 21 Review Session Oct 19 2pm in TH201 (that’s here)

Read this article for Friday [1]Chelazzi L, Miller EK, Duncan J, Desimone R. A neural basis for visual search in inferior temporal cortex. Nature 1993;

Attention as Information Selection. Early Selection Early Selection model postulated that attention acted as a strict gate at the lowest levels of sensory.

Multisensory convergence Traditionally, represented by inputs from more than one sensory modality, each of which can independently activate the target.

Brodmann Areas Rizzolatti et al., Electroencephalography and clinical Neurophysiology, 1998 Matelli Areas.

Attention II Selective Attention & Visual Search.

Attention II Theories of Attention Visual Search.

Audiovisual Multisensory Facilitation: A Fresh Look at Neural Coactivation and Inverse Effectiveness. Lynnette Leone North Dakota State University.

An Electrophysiological study of translation priming in French/English bilinguals Katherine J. Midgley 1,2, Jonathan Grainger 2 & Phillip J. Holcomb 1.

Studying Visual Attention with the Visual Search Paradigm Marc Pomplun Department of Computer Science University of Massachusetts at Boston

Experimental study of morphological priming: evidence from Russian verbal inflection Tatiana Svistunova Elizaveta Gazeeva Tatiana Chernigovskaya St. Petersburg.

Major Sensory and Perceptual Systems SenseSource of information SeeingLight HearingSound BalanceGravity and acceleration TouchPressureTemperature PoseJoint.

Studying Memory Encoding with fMRI Event-related vs. Blocked Designs Aneta Kielar.

Right hemisphere sensitivity to word & sentence level context: Evidence From Event-Related Brain Potentials. Coulson, S. Federmeier, K.D., Van Petten,

Understanding Action Verbs- Embodied Verbal Semantics Approach Pavan Kumar Srungaram M.Phil Cognitive Science (09CCHL02) Supervisor: Prof. Bapi.

Psych 435 Attention. Issues Capacity –We can’t respond to everything in the environment –Too many pieces of information –we can only actively respond.

Results Introduction Nonconditional Feedback Selectively Eliminates Conflict Adaption Summary Methods 38 participants performed a parity judgment task.

# Attentional Volleying Across Visual Quadrants Andrew S. Clement 1,2 & Nestor Matthews 1 1 Department of Psychology, Denison University, 2 Department.

The effects of working memory load on negative priming in an N-back task Ewald Neumann Brain-Inspired Cognitive Systems (BICS) July, 2010.

Multisensory processing: an introduction

Psych 335 Attention. Issues Capacity –We can’t respond to everything in the environment –Too many pieces of information –we can only actively respond.

Bayesian inference accounts for the filling-in and suppression of visual perception of bars by context Li Zhaoping 1 & Li Jingling 2 1 University College.

Multimodal Virtual Environments: Response Times, Attention, and Presence B 陳柏叡.

8. What are the advantages and disadvantages of using a virtual reality environment to study the brain and behavior? 9.Give examples of the way that virtual.

Orienting Attention to Semantic Categories T Cristescu, JT Devlin, AC Nobre Dept. Experimental Psychology and FMRIB Centre, University of Oxford, Oxford,

Introduction Ruth Adam & Uta Noppeney Max Planck Institute for Biological Cybernetics, Tübingen Scientific Aim Experimental.

Multimodal Perception

ANT Z=52 R ACUE - PASSIVE VCUE - PASSIVE 1300 msVoltageCSD.31uV.03uV/cm 2 AIM We investigate the mechanisms of this hypothesized switch-ERP.

Selective Attention

How Does Multiple Group Membership Affect Face Recognition in Asian Participants? Sarah Pearson, Jane Farrell, Christopher Poirier, and Lincoln Craton.

Ch. 13 A face in the crowd: which groups of neurons process face stimuli, and how do they interact? KARI L. HOFFMANN 2009/1/13 BI, Population Coding Seminar.

Chapter 5 Short-Term and Working Memory. Some Questions to Consider Why can we remember a telephone number long enough to place a call, but then we forget.

Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors W. David Hairston, Ph.D Advanced Neuroscience Imaging Research Lab.

Assist. Prof. Dr. Ilmiye Seçer Fall

Development of Audiovisual Integration in Central and Peripheral Vision Yi-Chuan Chen, Terri L. Lewis, David I. Shore, and Daphne Maurer Department.

Visual Memory is Superior to Auditory Memory

Models of substitution masking

Multisensory perception: Beyond modularity and convergence

The Development of Emotional Interactions Across the Senses:

The Nose Smells What the Eye Sees

Investigating the Attentional Blink With Predicted Targets

تلفيق اطلاعات سنسوري: مکانيسم و مدلها

Bayesian inference Li Zhaoping1 & Li Jingling2 accounts for the

A Vestibular Sensation: Probabilistic Approaches to Spatial Perception

Multisensory Perception and Biological Motion

Multisensory integration: perceptual grouping by eye and ear

Feature-based attention

Multisensory Integration and Attention in Developmental Dyslexia

same area of visual space – same region of the brain

Multisensory Integration: Maintaining the Perception of Synchrony

Volume 22, Issue 5, Pages (March 2012)

Presentation transcript:

The role of auditory-visual integration in object recognition Clara Suied 1, Nicolas Bonneel 2 and Isabelle Viaud-Delmon 1 1 CNRS – UPMC UMR 7593 Hôpital de la Salpêtrière, Paris, France 2 REVES / Inria Sophia-Antipolis, France Acoustics’08 Research supported by the EU IST FP6 Open FET project CROSSMOD

Recognition of natural object Recognizing a natural object involves pooling information from various sensory modalities And to ignore information from competing objects How do these multisensory information interact to form a unique object concept?

Object recognition To direct action to objects, spatial information needs to be encoded and this might interact with object perception –For recognition tasks with the spatial dimension not relevant to the task, conflicting results (Gondan et al., 2005; Teder-Salerjarvi et al., 2005) Realistic object are of interest in the study of multisensory integration, since a given object can be identified through any of several single modalities –Little behavioural studies with realistic objects (e.g. Molholm et al., 2004 for an ERP study; Laurienti et al., 2004 for linguistic-type stimuli)

Identification task: go/no-go –When the target (telephone) is either heard or seen, press the button as fast as possible –Withold response when distractor (frog) is presented alone Main experiment: Object Recognition

Go conditions A+ 40 RING A+ 0 V+ Unimodal RING A+ 0 V+ A+ 40 V+ Bimodal semantically congruent CROAK RING A+ 0 V- A- 0 V+A- 40 V+A+ 40 V- Bimodal semantically incongruent

No-Go conditions A- 40 CROAK A- 0 V- Unimodal A- 0 V- A- 40 V- Bimodal semantically congruent CROAK

Experimental questions Spatial alignment necessary for fast object recognition? Larger auditory-visual integration for realistic objects? Effect of distractors (semantic congruence) on performance?

Results BimodalVisual targetAuditory target

Experimental questions Spatial alignment necessary for fast object recognition? Larger auditory-visual integration for realistic objects? Effect of distractors (semantic congruence) on performance?

Spatial alignment BimodalVisual targetAuditory target

2 (spatial alignment) x 4 (conditions) repeated-measures ANOVA Main effect of the spatial alignment (F 1,19 =17.68; p<0.0005) Main effect of the condition (F 3,57 =65.36; ε= 0.8; p<0.0001) But NO INTERACTION  the spatial effect is a Stimulus-Response Compatibility (Simon and Craft, 1970; Simon et al., 1981; Lu and Proctor, 1995) Spatial alignment Spatial alignment does not facilitate object recognition

Experimental questions Spatial alignment necessary for fast object recognition? Larger auditory-visual integration for realistic objects? Effect of distractors (semantic congruence) on performance?

Auditory-visual integration

p < Auditory-visual integration

AV integration and not statistical facilitation Race Model (Miller, 1982)

Computation of the effect size of the AV integration observed in the A+ 0 V+ condition (Cohen’s d; Cohen, 1988) Comparison with the size of AV integration previously observed in the literature Size of the AV integration where

Large AV integration

Experimental questions Spatial alignment necessary for fast object recognition? Larger auditory-visual integration for realistic objects? Effect of distractors (semantic congruence) on performance?

p < Role of a distractor on object recognition Auditory distractorVisual distractor

When the distractor is visual –No performance cost when processing an auditory target When the distractor is auditory –There is a performance cost when processing a visual target It seems impossible to ignore an auditory distractor Role of a distractor on object recognition

Conclusion Large bimodal integration effect –Size of the visual object, realism, 3D and large display, immersive No effect of spatial alignment on object recognition –Spatial alignment important for saccade generation or signal detection (Stein and Meredith, 1993; Hughes et al., 1994; Frens et al., 1995; Harrington and Peck, 1998) –Object recognition is a function where spatial alignment is not essential It could reflect the fact that this function probably involves brain regions containing neurons with broad spatial receptive fields A possible asymmetry in the attentional filtering of irrelevant auditory and visual information –Similar asymmetry for cueing effect in detection tasks (Schmitt et al., 2000) –Alerting role of the auditory system?

AV integration and not statistical facilitation Race Model (Miller, 1982)