1. Visually-induced auditory spatial adaptation in monkeys and humans Norbert Kopčo, I-Fan Lin, Barbara Shinn-Cunningham, Jennifer Groh Center for Cognitive Neuroscience, Duke University Hearing Research Center, Boston University Technical University, Košice, Slovakia

2. 2Nov 6, 2007SFN 07 San Diego Introduction Vision affects auditory spatial perception (e.g. the ventriloquist effect) Visually-induced shifts in sound localization can persist in the absence of visual stimuli (e.g. barn owl prism adaptation studies)

3. 3Nov 6, 2007SFN 07 San Diego QUESTION How does vision calibrate sound perception in primates? - monkeys and humans Unlike barn owls, monkeys and humans make eye movements. With every eye movement, the relationship between visual space and auditory space changes.

4. 4Nov 6, 2007SFN 07 San Diego Visual and auditory spatial information are different! VISION: Retina provides “map” of object locations Locations shift when eyes move Frame of reference is “eye- centered”

5. 5Nov 6, 2007SFN 07 San Diego Visual and auditory spatial information are different! AUDITORY: Sound location calculated from interaural timing and level differences Cue values do NOT shift when eyes move Frame of reference is “head-centered”

6. 6Nov 6, 2007SFN 07 San Diego Goals Does visual-calibration of auditory space occur in eye-centered, head-centered, or a hybrid coordinate system? Are humans and monkeys similar?

7. 7Nov 6, 2007SFN 07 San Diego Experimental Setup Audiovisual display as viewed by the subject Horizontal location (degrees) Vertical location (degrees) 9 speakers in front of listener (~1 m distance), separated by 7.5° (humans) or 6° (monkeys) Light-emitting diodes (LEDs) at three center speakers: - aligned with speakers, or - displaced to the left or to the right (by 5°-humans, 6°-monkeys) 2 LEDs below speaker array used as fixation points (FP) Stimuli: Auditory stimulus: 300-ms broadband noise burst Audio-Visual stimulus: Same noise with synchronously lid LED.

8. 8Nov 6, 2007SFN 07 San Diego Experiment: Data Presentation Format Sample Stimulus-Response Raw Data Plot Actual Stimulus Location (°) Perceived / Responded Location (°) Sample Plot Showing Bias in Responses (Response – Actual Location) Actual Stimulus Location (°) Bias in Responses (°) Rightward bias Leftward bias Rightward bias Leftward bias

9. 9Nov 6, 2007SFN 07 San Diego Experiment: Hypothesis 1. Induce Ventriloquist Effect for AV stimuli presented in central sub- region of space, keeping fixation point on the right ( +8°) 2. Test that adaptation results in the effect being present also for Auditory-only stimuli (same FP). 3. Induced shift will decrease outside the trained sub-region (for the same FP). 4. Move FP to the left. Test effect on Auditory-only stimuli: - No change  Head-centered. - Response pattern moves with eyes  Eye-centered. - Combined representations. Stimulus Location (°) Bias (°) Audiovisual display Expected Responses FP LEDs Speakers

10. 10Nov 6, 2007SFN 07 San Diego Experiment: Procedure Audiovisual display as viewed by the subject Horizontal location (degrees) Vertical location (degrees) One trial consists of: 1. Fixation point (FP) appears. 2. Subject fixates FP. 3. Target stimulus is presented (Audio-Visual or Auditory-only). 4. Subject saccades to perceived location of stimulus (humans instructed to always saccade to sound). 5. Monkeys only: Reward for responding within a criterion window (+- 10° from speaker).

11. 11Nov 6, 2007SFN 07 San Diego Experiment: Procedure Experiment divided into 1-hour blocks (12 for humans, 16 for monkeys; 7 humans, 2 monkeys). Within a block three types of trials, randomly interleaved: Three types of AV stimuli (AV stimulus type kept constant within a block. Data also collected with AV FP on left. These data mirror-flipped to simplify presentation. For monkeys, also AV- aligned stimuli at +-30° to enforce non-linearity): AV stimuli: 50 % FP LEDs Speakers A-only,trained FP: 25%A-only,shifted FP: 25% AV stimuli aligned Hypometric shift Hypermetric shift

12. 12Nov 6, 2007SFN 07 San Diego Results: Humans Stimulus Location (°) Audiovisual display Expected Responses FP LEDs Speakers Human Behavior Data collapsed across direction of induced shift AV responses: - as expected Trained FP A-only responses: - Shift induced in trained sub-region - Generalization to untrained regions (asymmetrical) Shifted FP A-only responses: - Shift reduced in center region Head-centered repre- sentation, modulated by eye position Mean+SE

13. 13Nov 6, 2007SFN 07 San Diego Results: Humans vs. Monkeys Audiovisual display FP LEDs Speakers Human Behavior Monkey data (only hypometric) AV responses: - as expected Trained FP A-only responses: - Shift in trained sub- region weaker - Generalization to untrained regions stronger (asymmetry opposite to humans) Shifted FP A-only responses: - Shifted with eyes Representation more eye-centered Mean+SE Monkey Behavior Mean 6 6 -24 0 24 Mean+individuals Mean+SE Mean+individuals

14. 14Nov 6, 2007SFN 07 San Diego Summary Good news – the main results are consistent across species: Locally induced ventriloquist effect results in short-term adaptation, causing shifts in responses to A-only stimuli from trained sub-region. The pattern of induced shift is modified as the eyes move. Bad news – there is a lot of differences between species: HumansMonkeys Representationhead-centered, eye-modulatedeye-centered Generalization to untrained sub-regionsmore on the side away from FPopposite Difference between hyper- and hypometric shiftsnoyes Representation when shift induced on side (data not shown)head-centered, no eye modulationeye-centered

15. 15Nov 6, 2007SFN 07 San Diego Discussion The differences between species can be caused by: 1. differences in neural representation and/or learned behavior E.g., monkeys often respond by double saccades 2. differences in the stage along the pathway at which the calibration occurs (Insert figure of ear->CN->SOC->IC->MGB->Aud. Ctx->Parietal Ctx->SC->oculomotor) E.g., in humans, the head-centered shift was induced fast, while the eye-centered modulation was much slower  different time scales may imply adaptation at different stages 3. procedural differences (e.g., monkeys work for water) Future work: Humans: examine temporal and spatial factors influencing the eye-centered modulation. Monkeys: attempt to induce stronger effect using larger A-V separation.