Audiovisual Multisensory Facilitation: A Fresh Look at Neural Coactivation and Inverse Effectiveness. Lynnette Leone North Dakota State University

“However, the primary somatic, visual and auditory cortices are not interconnected, and each projects to very restricted and entirely separate fields chiefly in their immediate vicinity…” (Jones and Powell, 1970) “Such integration is as critical for making sense of the inputs the brain receives from different modalities as it is for interpreting multiple inputs from any single modality…for the brain to integrate them [these inputs] the different senses must ultimately have access to the same neurons.” (Stein and Meredith 1993)

Outline 1.Multisensory Integration 2.Redundant Signals Effect 3. Inverse Effectiveness 4. The Current Study 5. Future Directions

Multisensory integration Definition : (MI) refers to the process by which convergence of information from two or more individual sensory systems onto particular neurons results in a neuronal response that is qualitatively and quantitatively different than the responses of these neurons to individual signals (Calvert, 2001).

Multisensory integration McGurk Effect Visual Rabbit

Multisensory integration Facilitation – more likely to occur when two things happen at the same time and / or in the same place. Suppression – more likely to occur if two event occur at widely different times and / or places.

Redundant Signals Effect (RSE) Definition: the modulation of reaction time to pairings of sensory stimuli presented simultaneously over one or more sensory channels. - facilitative MI - not exclusively multisensory

Redundant Signals Effect (RSE) Separate Activation vs. Neural Coactivation - RACE models – Miller’s Inequality P (RT < t|A and V) ≤ [ P (RT < t|A) + P (RT < t|V)] - [P (RT < t|A) ٭ P (RT < t|V)]

Redundant Signals Effect (RSE) Some studies: Miller, J. Cognitive Psychology (1982) Experiment 1 Subjects: 74, undergrads rt handed Visual Stimuli: asterisk ( ٭ ) Auditory Stim: 780 Hz tone (150ms) Task: Simple RT 3 conditions: A, V, AV Randomly Interleaved

Redundant Signals Effect (RSE) Miller, J. Cognitive Psychology, (1982) Results - mean RT V = 412 ms, A = 409 ms, AV = 326 ms - violations of inequality occurred across a range of reaction times (250 – 350 ms)

Redundant Signals Effect (RSE) Molholm et al Cognitive Brain Research (2002) Subjects: 12 (5 female, 11 RH), 23.8±2.7 yr Visual Stimuli: 60 ms, 1.2 deg disc Auditory Stim: 1000 Hz, 60 ms, 75 dB SPL Task: Simple RT, right index finger button press 3 conditions: A, V, AV Randomly Interleaved ISI: ms

Redundant Signals Effect (RSE) Molholm, S. et al Cognitive Brain Research (2002)

Redundant Signals Effect (RSE) Spatial and temporal constraints - Miller, (1986) - Stein et al, (1996) - Frassinetti et al, (2002)

Inverse Effectiveness Rule: combinations of weaker stimuli lead to greater facilitative effects. Studies: - Stein and Meredith, Diederich and Colonius, Holmes, 2007

Current Study A.How might changes in the processing time of one system influence the time- course of the RSE? - increasing stimulus contrast leads to decreases in reaction time. (Harworth and Levi, 1978, Murray and Plainis, 2003 and Vassilev, Mihaylova and Bonnet, 2002)

Current Study B. Hypothesis – If neural coactivation is indeed responsible for the RSE, then changes in signal processing time in one modality will necessarily change the facilitative effects obtained when those signals are paired with signals from another modality.

Method Pretest Procedure: - Single-interval forced choice signal detection paradigm YesNo YesHitsFalse alarms NoMisses Correct rejections Signal Response

Method Pretest Procedure: - 15 Blocks; each block contained 24 unisensory visual (2 x 12 contrast intensities, spatial frequency 1 c/d) and 24 unisensory auditory (2 x 12 levels of dB attenuation), and 2 catch trials. - d’ calculated for responses to individual stimuli (d’ = Z(h) – Z (FA); nonlinear (sigmoid) least- squares regression interpolated stimulus intensities yielding criterion performance (d’ = 2).

Visual stimulus (Gabor patch = 1 c/d; duration = 100ms; view distance = 114 cm, 2.25 ◦ eccentricity from fixation ) Auditory Stimulus (1000 Hz pure tone duration = 100ms) Subjects: 7 (4 Male, 7 RH); 23 – 55 yrs

Method Experimental Procedure: - Single-interval signal detection paradigm; - Each block: randomly interleaved trials of unisensory visual stimuli, unisensory auditory stimuli, catch trials, and audiovisual multisensory combinations at stimulus onset asynchronies ranging from -100 to +200 ms. - Task: Subjects instructed to respond as quickly and accurately as possible when they perceived a stimulus. Calculated d’ for unisensory stimuli every 2 blocks and adjusted intensity settings to ensure criterion response accuracy (d’ = 2).

Data Analysis

Experiment 1: Results

Experiment 2: Paradigm replicated experiment 1 except that visual stimulus contrast was increased (3x).

Experiment 3: Paradigm replicated experiment 1 except that auditory stimulus volume increased (3x).

Experiment 4: Paradigm replicated experiment 1 except that both visual stimulus contrast and auditory stimulus volume were increased (3x).

Conclusions What the heck? Attentional effects: - endogenous attention - exogenous attention

What’s next Experiment 5 - examines attentional contributions using a variation of a Posner cuing paradigm to separate the effects of attention from those of neuronal interaction. Experiment 6 - examines inverse effectiveness gradient.

Future Directions Dark adaptation - rods dominate our vision in the dark - rods operate on the order of 100ms slower than cones ERP

Acknowledgements: Dr. Mark McCourt Dr. Wolfgang Teder - SäleJärvi Tech Support: Dan Gu, Brian Pasieka All of the subjects (most of whom are in this room )