Visually-induced auditory spatial adaptation in monkeys and humans Norbert Kopčo, I-Fan Lin, Barbara Shinn-Cunningham, Jennifer Groh Center for Cognitive Neuroscience, Duke University Hearing Research Center, Boston University Technical University, Košice, Slovakia

2Nov 6, 2007SFN 07 San Diego Introduction Vision affects auditory spatial perception (e.g. the ventriloquist effect) Visually-induced shifts in sound localization can persist in the absence of visual stimuli (e.g. barn owl prism adaptation studies)

3Nov 6, 2007SFN 07 San Diego QUESTION How does vision calibrate sound perception in primates? - monkeys and humans Unlike barn owls, monkeys and humans make eye movements. With every eye movement, the relationship between visual space and auditory space changes. Visual and auditory spatial information are different!

4Nov 6, 2007SFN 07 San Diego Visual and auditory spatial information are different! VISION: Retina provides “map” of object locations Locations shift when eyes move Frame of reference is “eye- centered”

5Nov 6, 2007SFN 07 San Diego Visual and auditory spatial information are different! AUDITORY: Sound location calculated from interaural timing and level differences Cue values do NOT shift when eyes move Frame of reference is “head- centered”

6Nov 6, 2007SFN 07 San Diego Goals Perform behavioral experiments to answer the following questions: Does visual calibration of auditory space occur in eye-centered, head-centered, or a hybrid coordinate system? Are humans and monkeys similar?

7Nov 6, 2007SFN 07 San Diego Experimental Setup Audiovisual display as viewed by the subject Horizontal location (degrees) Vertical location (degrees) 9 speakers in front of listener (~1 m distance), separated by 7.5° (humans) or 6° (monkeys) Light-emitting diodes (LEDs) at three center speakers: - aligned with speakers, or - displaced to the left or to the right (by 5°-humans, 6°-monkeys) 2 LEDs below speaker array used as fixation points (FP) Stimuli: Auditory stimulus: 300-ms broadband noise burst Audio-Visual stimulus: Same noise with synchronously lid LED.

8Nov 6, 2007SFN 07 San Diego Rationale and Hypothesis 1. Induce Ventriloquist Effect for AV stimuli presented in central sub- region of space, keeping fixation point on the right ( +11°) 2. Test that adaptation results in the effect being present also for Auditory-only stimuli (same FP). 3. Induced shift will decrease outside the trained sub-region (for the same FP). 4. Move FP to the left. Test effect on Auditory-only stimuli: - No change  Head-centered. - Response pattern moves with eyes  Eye-centered. - Combined representations. Stimulus Location (°) Magnitude (°) Audiovisual display Expected Responses FP LEDs Speakers

9Nov 6, 2007SFN 07 San Diego Experiment: Procedure Audiovisual display as viewed by the subject Horizontal location (degrees) Vertical location (degrees) One trial consists of: 1. Fixation point (FP) appears. 2. Subject fixates FP. 3. Target stimulus is presented (Audio-Visual or Auditory-only). 4. Subject saccades to perceived location of stimulus (humans instructed to always saccade to sound). 5. Monkeys only: Reward for responding within a criterion window (+- 10° from speaker).

10Nov 6, 2007SFN 07 San Diego Experiment: Procedure Experiment divided into 1-hour blocks. 12 blocks for humans, 16 for monkeys. Subjects: 7 humans, 2 monkeys. Within a block three types of trials, randomly interleaved: AV stimuli 50 % of trials FP LEDs Speakers A-only stimuli trained FP: 25% of trials A-only stimuli shifted FP: 25% of trials AV stimulus type kept constant within a block. Three types of AV stimulus: 1) aligned. 2) displaced hypermetrically. 3) displaced hypometrically.

11Nov 6, 2007SFN 07 San Diego Results: Humans Audiovisual display Expected Responses FP LEDs Speakers Stimulus Location (°) Magnitude of Induced Shift (°) or Trained FP A-only responses: - Shift induced in trained sub-region - Generalization to untrained regions (asymmetrical) Shifted FP A-only responses: - Shift reduced in center region Head-centered representation, modulated by eye position

12Nov 6, 2007SFN 07 San Diego Results: Monkeys Audiovisual display Humans: FP LEDs Speakers Stimulus Location (°) Magnitude of Induced Shift (°) Trained FP A-only responses: - Shift in trained sub- region weaker - Generalization to untrained regions stronger (asymmetry opposite to humans) Shifted FP A-only responses: - Shifted with eyes Representation more mixed

13Nov 6, 2007SFN 07 San Diego Summary The main results are consistent across species: Locally induced ventriloquist effect results in short-term adaptation, causing shifts in responses to A-only stimuli from trained sub-region. The induced shift generalizes outside the trained sub-region, with gradually decreasing strength (However, the pattern of generalization differs across the species) The pattern of induced shift changes as the eyes move. But, overall, it appears to be in a representation frame that is more head-centered than eye-centered.

14Nov 6, 2007SFN 07 San Diego Discussion Posterior Parietal Cortex Neural adaptation could have been induced at several stages along the Pathway (IC, MGB, AC, PPX, MC, SC) In humans, multiple effects observed at different temporal scales  likely adaptation at multiple stages Future work Extend Examine temporal and spatial factors influencing the eye-centered modulation. Midbrain Pons Cerebrum Thalamus Midbrain Pons Thalamus

15Nov 6, 2007SFN 07 San Diego Summary and discussion the main results are consistent across species: Locally induced ventriloquist effect results in short-term adaptation, causing shifts in responses to A-only stimuli from trained sub-region. The pattern of induced shift is modified as the eyes move. Bad news – there is a lot of differences between species: HumansMonkeys Representationhead-centered, eye-modulatedeye-centered Generalization to untrained sub-regionsmore on the side away from FPopposite Difference between hyper- and hypometric shiftsnoyes Representation when shift induced on side (data not shown)head-centered, no eye modulationeye-centered

16Nov 6, 2007SFN 07 San Diego Results: Humans Audiovisual display Expected Responses FP LEDs Speakers Human Behavior Mean+SE Stimulus Location (°) Magnitude of Induced Shift (°)

17Nov 6, 2007SFN 07 San Diego Results: Humans Stimulus Location (°) Audiovisual display Expected Responses FP LEDs Speakers Human Behavior Data collapsed across direction of induced shift Trained FP A-only responses: - Shift induced in trained sub-region - Generalization to untrained regions (asymmetrical) Shifted FP A-only responses: - Shift reduced in center region Head-centered repre- sentation, modulated by eye position Mean+SE

18Nov 6, 2007SFN 07 San Diego Results: Humans vs. Monkeys Audiovisual display FP LEDs Speakers Human Behavior Monkey data (only hypometric) AV responses: - as expected Trained FP A-only responses: - Shift in trained sub- region weaker - Generalization to untrained regions stronger (asymmetry opposite to humans) Shifted FP A-only responses: - Shifted with eyes Representation more eye-centered Mean+SE Monkey Behavior Mean Mean+individuals Mean+SE Mean+individuals