Visual Adaptation And Spatial Auditory Processing Peter Lokša, Norbert Kopčo Collaborators: Jenni Groh, Barb Shinn-Cunningham, I-Fan Lin

2 Introduction Visual stimuli can affect the perception of sound location e.g. the Ventriloquism Effect Way to go Red Sox! Way to go Red Sox! But does effect persist? - barn owls: prism adaptation (Knudsen et al.) - monkeys: “ventriloquism aftereffect” (Woods and Recanzone, Curr. Biol. 2004) e.g. the Ventriloquism Afterffect

3 What is reference frame Eye-centered? Head (ear) -centered? Oculomotor? ? ? What is the reference frame?

4 Methods Basic idea: 1. Pre-adaptation baseline: Measure auditory saccade accuracy 2. Adaptation phase: Present combined visual-auditory stimuli, with visual location shifted 3. Compare auditory saccade accuracy pre- and post-adaptation

5 Method: center Audiovisual display Expected behavior Stimulus Location (°) Magnitude (°) Fix head to face 0° Induce shift: - in only one region of space - from a single fixation point Test to see if shift generalizes to the same sub-region in: - head-centered space - eye-centered space Experiment divided into 1-hour blocks: (12 for humans,16 for monkeys) Within a block, 3 types of randomly interleaved trials: - Training AV stim: 50% - Test A-only stimuli: 50% (25% from trained, 25% shifted FP) FP LEDs Speakers

6 Follow-up Eye-centered? Head (ear) -centered? Oculomotor? ? ? Are results independent of where ventriloquism is introduced? - Spatial auditory resolution is lower in the periphery - Maier et al – binaural cue adaptation only in center

7 Method: periphery Expected behavior Stimulus Location (°) Magnitude (°) Fix head to face 0° Induce shift: - in only one region of space - from a single fixation point Test to see if shift generalizes to the same sub-region in: - head-centered space - eye-centered space Experiment divided into 1-hour blocks: (12 for humans,16 for monkeys) Within a block, 3 types of randomly interleaved trials: - Training AV stim: 50% - Test A-only stimuli: 50% (25% from trained, 25% shifted FP) Audiovisual display FP LEDs Speakers

8 Results: AV-shifted data

Results: AV-aligned Baseline Humnas: Monkeys:

10 Summary The main results are consistent across species and regions: - it’s possible to locally introduce ventriloquism aftereffect - the reference frame is a mixture between head- and eye-centered But Visual stimuli influence saccade responses to auditory stimuli even when the AV-stimuli are spatially aligned: - Expanding the auditory spatial map outside the trained area - In an head-centered way when AV training is in center - In a way that is modulated by eye-centered representation when AV training is in periphery

Collaborators Jenni Groh, Duke University I-Fan Lin, NTT Tokio Barb Shinn-Cunningham, BU