2. The role of auditory-visual integration in object recognition Clara Suied 1, Nicolas Bonneel 2 and Isabelle Viaud-Delmon 1 1 CNRS – UPMC UMR 7593 Hôpital de la Salpêtrière, Paris, France 2 REVES / Inria Sophia-Antipolis, France Acoustics’08 Research supported by the EU IST FP6 Open FET project CROSSMOD

3. Recognition of natural object Recognizing a natural object involves pooling information from various sensory modalities And to ignore information from competing objects How do these multisensory information interact to form a unique object concept?

4. Object recognition To direct action to objects, spatial information needs to be encoded and this might interact with object perception –For recognition tasks with the spatial dimension not relevant to the task, conflicting results (Gondan et al., 2005; Teder-Salerjarvi et al., 2005) Realistic object are of interest in the study of multisensory integration, since a given object can be identified through any of several single modalities –Little behavioural studies with realistic objects (e.g. Molholm et al., 2004 for an ERP study; Laurienti et al., 2004 for linguistic-type stimuli)

5. Identification task: go/no-go –When the target (telephone) is either heard or seen, press the button as fast as possible –Withold response when distractor (frog) is presented alone Main experiment: Object Recognition

7. Go conditions A+ 40 RING A+ 0 V+ Unimodal RING A+ 0 V+ A+ 40 V+ Bimodal semantically congruent CROAK RING A+ 0 V- A- 0 V+A- 40 V+A+ 40 V- Bimodal semantically incongruent

8. No-Go conditions A- 40 CROAK A- 0 V- Unimodal A- 0 V- A- 40 V- Bimodal semantically congruent CROAK

24. Experimental questions Spatial alignment necessary for fast object recognition? Larger auditory-visual integration for realistic objects? Effect of distractors (semantic congruence) on performance?

25. Results BimodalVisual targetAuditory target

26. Experimental questions Spatial alignment necessary for fast object recognition? Larger auditory-visual integration for realistic objects? Effect of distractors (semantic congruence) on performance?

27. Spatial alignment BimodalVisual targetAuditory target

28. 2 (spatial alignment) x 4 (conditions) repeated-measures ANOVA Main effect of the spatial alignment (F 1,19 =17.68; p<0.0005) Main effect of the condition (F 3,57 =65.36; ε= 0.8; p<0.0001) But NO INTERACTION  the spatial effect is a Stimulus-Response Compatibility (Simon and Craft, 1970; Simon et al., 1981; Lu and Proctor, 1995) Spatial alignment Spatial alignment does not facilitate object recognition

29. Experimental questions Spatial alignment necessary for fast object recognition? Larger auditory-visual integration for realistic objects? Effect of distractors (semantic congruence) on performance?

30. Auditory-visual integration

31. p < 0.0001 Auditory-visual integration

32. AV integration and not statistical facilitation Race Model (Miller, 1982)

33. Computation of the effect size of the AV integration observed in the A+ 0 V+ condition (Cohen’s d; Cohen, 1988) Comparison with the size of AV integration previously observed in the literature Size of the AV integration where

34. Large AV integration

35. Experimental questions Spatial alignment necessary for fast object recognition? Larger auditory-visual integration for realistic objects? Effect of distractors (semantic congruence) on performance?

36. p < 0.005 Role of a distractor on object recognition Auditory distractorVisual distractor

37. When the distractor is visual –No performance cost when processing an auditory target When the distractor is auditory –There is a performance cost when processing a visual target It seems impossible to ignore an auditory distractor Role of a distractor on object recognition

38. Conclusion Large bimodal integration effect –Size of the visual object, realism, 3D and large display, immersive No effect of spatial alignment on object recognition –Spatial alignment important for saccade generation or signal detection (Stein and Meredith, 1993; Hughes et al., 1994; Frens et al., 1995; Harrington and Peck, 1998) –Object recognition is a function where spatial alignment is not essential It could reflect the fact that this function probably involves brain regions containing neurons with broad spatial receptive fields A possible asymmetry in the attentional filtering of irrelevant auditory and visual information –Similar asymmetry for cueing effect in detection tasks (Schmitt et al., 2000) –Alerting role of the auditory system?

40. AV integration and not statistical facilitation Race Model (Miller, 1982)