Visually-induced auditory spatial adaptation in monkeys and humans Norbert Kopčo, I-Fan Lin, Barbara Shinn-Cunningham, Jennifer Groh Center for Cognitive Neuroscience, Duke University Hearing Research Center, Boston University Technical University, Košice, Slovakia Barb suggested making the coordinate-frame animation more obvious Lid -> Lit the data are all relative to the no-adaptation baselines. Those things just didn't come across in an obvious way. Possible reasons for differences: Monkey don’t try to look at sounds 1

Introduction Visual stimuli can affect the perception of sound location e.g. the Ventriloquism Effect Way to go Red Sox! Way to go Red Sox! ARO abstract: Visual calibration of auditory spatial perception in humans and monkeys Norbert Kopco, I-Fan Lin, Barbara Shinn-Cunningham, Jennifer Groh Duke, BU When you see and hear something simultaneously, and you will often perceive the sound as coming from the same location as the visual stimulus. This is known as the ventriloquist effect. Indeed, the influence of visual stimuli can often outlast the visual stimulus itself, causing persistent changes in where sounds are perceived to be, even when the sounds are presented by themselves. The neural mechanisms underlying such visual capture of auditory spatial information have been extensively studied in barn owls using a prism adaptation paradigm, but comparatively little is known about the process either perceptually or at the neurophysiological level, in primates. In this study, I will describe our recent experiments on the nature of visual induced-recalibration of sound location in humans and monkeys. But does effect persist? Nov 6, 2007 SFN 07 San Diego

Introduction Visual stimuli can affect the perception of sound location e.g. the Ventriloquism Effect Way to go Red Sox! ARO abstract: Visual calibration of auditory spatial perception in humans and monkeys Norbert Kopco, I-Fan Lin, Barbara Shinn-Cunningham, Jennifer Groh Duke, BU When you see and hear something simultaneously, and you will often perceive the sound as coming from the same location as the visual stimulus. This is known as the ventriloquist effect. Indeed, the influence of visual stimuli can often outlast the visual stimulus itself, causing persistent changes in where sounds are perceived to be, even when the sounds are presented by themselves. The neural mechanisms underlying such visual capture of auditory spatial information have been extensively studied in barn owls using a prism adaptation paradigm, but comparatively little is known about the process either perceptually or at the neurophysiological level, in primates. In this study, I will describe our recent experiments on the nature of visual induced-recalibration of sound location in humans and monkeys. But does effect persist? - barn owls: prism adaptation (Knudsen et al.) - monkeys: “ventriloquism aftereffect” (Woods and Recanzone, Curr. Biol. 2004) Nov 6, 2007 SFN 07 San Diego

GOALS Ventriloquism “aftereffect” in saccade task, in monkeys and humans? - well-defined sensory-motor paradigm - bridge to barn owl prism adaptation studies (on different time scale) Reference frame of plasticity? - Visual, auditory, or oculomotor reference frame? Nov 6, 2007 SFN 07 San Diego

Methods Basic idea: 1. Pre-adaptation baseline: Measure auditory saccade accuracy 2. Adaptation phase: Present combined visual-auditory stimuli, with visual location shifted 3. Compare auditory saccade accuracy pre- and post-adaptation Nov 6, 2007 SFN 07 San Diego

Q1: Does it work? Initial experiment Design: Monkey Pre-adaptation baseline – ~100 Auditory-only trials Adaptation phase – 80% V-A stimuli, visual stimulus shifted 6 deg. Left or Right 20% Auditory-only Compare Auditory-only trials from adaptation phase to pre- adaptation phase Sounds: Loudspeakers Visual stimuli: LEDs Nov 6, 2007 SFN 07 San Diego

RESULTS Nov 6, 2007 SFN 07 San Diego

RESULTS Nov 6, 2007 SFN 07 San Diego

RESULTS Nov 6, 2007 SFN 07 San Diego

Q2: reference frame 2a. What is the reference frame? Eye-centered? ? ? 2b. Is the reference frame the same for humans and monkeys? Eye-centered? ? ? Oculomotor? Head (ear) -centered? Nov 6, 2007 SFN 07 San Diego

Method Fix head to face 0° Induce shift: - in only one region of space - from a single fixation point Test to see if shift generalizes to the same sub-region in: - head-centered space - eye-centered space Experiment divided into 1-hour blocks: (12 for humans, 16 for monkeys) Within a block, 3 types of randomly interleaved trials: - Training AV stim: 50% - Test A-only stimuli: 50% (25% from trained, 25% shifted FP) Audiovisual display Expected behavior Stimulus Location (°) Magnitude (°) Speakers LEDs FP Nov 6, 2007 SFN 07 San Diego

Results: Humans Head-centered representation, Audiovisual display Trained FP A-only responses: - Shift induced in trained sub-region - Generalization to untrained regions (asymmetrical) FP LEDs Speakers Magnitude of Induced Shift (°) Shifted FP A-only responses: - Shift reduced in center region After inducing the expected shift in AV trials Stimulus Location (°) Expected Responses Head-centered or Eye-centered Head-centered representation, modulated by eye position Nov 6, 2007 SFN 07 San Diego

Magnitude of Induced Shift (°) Results: Monkeys Audiovisual display Trained FP A-only responses: - Shift in trained sub- region weaker - Generalization to untrained regions stronger (asymmetry oppo- site to humans) FP LEDs Speakers Magnitude of Induced Shift (°) Shifted FP A-only responses: - Shift decreases on the right - Shift increases on the left After inducing the expected shift in AV trials Stimulus Location (°) Expected Responses Humans: or Representation more mixed than in humans Nov 6, 2007 SFN 07 San Diego

Summary The main results are consistent across species: Locally induced ventriloquist effect results in short-term adaptation, causing 30-to-50% shifts in responses to A-only stimuli from trained sub-region. The induced shift generalizes outside the trained sub-region, with gradually decreasing strength (However, the pattern of generalization differs across the species) The pattern of induced shift changes as the eyes move. But, overall, it appears to be in a representation frame that is more head-centered than eye-centered. This shows that plasticty that underlies this adaptation There were differences Nov 6, 2007 SFN 07 San Diego

(Kandel, Schwartz, Jessel) and (Purves) Discussion Posterior Parietal Cortex Neural adaptation could have been induced at several stages along the pathway. Future work Examine temporal and spatial factors influencing the eye- centered modulation. Look at other trained sub-regions. Thank You! Support NIH Cerebrum Thalamus Thalamus Midbrain Midbrain Pons Pons Nov 6, 2007 SFN 07 San Diego (Kandel, Schwartz, Jessel) and (Purves)