The temporal resolution of binding brightness and loudness in dynamic random sequences Vision Sciences Society Annual Meeting 2012 Daniel Mann, Charles Chubb Department of Cognitive Sciences University of California, Irvine

Introduction In what ways can observers combine dynamic visual and auditory information (brightness & loudness)? Use a selective attention task to extract achievable attention filters for various brightness/loudness judgments Test the temporal limit of multimodal binding derived by Fujisaki & Nishida (2010) Briefly explain Nishida experiment. Goal 1: Test multiple attentional filters on brightness/loudness combinations and extract rules for the integration these cues Goal 2: Test the temporal limit of binding using dynamic random stimuli instead of the phase judgment stimuli used to define the limit (2-3Hz) by Fujisaki & Nishida (2010).

Background: Fujisaki & Nishida (2010) Phase judgment stimuli used to test cross modality binding temporal limit Result: Universal binding limit of 2-3Hz at 75% performance for all cross-modal stimuli tested

Methods: Stimuli Each stimulus was a quick randomly-ordered stream of 18 gray disks (83 ms per disk), each accompanied by a simultaneous burst of auditory white noise. Three levels of disk brightness and of noise loudness were used to produce 9 different types of audiovisual pairings (tokens). Sound presented in headphones. 0 to two because the lowest amplitude was actually blank/silent. Sound level measurements done with Bruel and Kjaer 2260 Investigator. A-weighted broadband detector with Fast time weighting. 5 bars of mac volume: 0 amplitude: ~33 dBSPL. Medium sound (.25) = 69 dBSPL. Loud sound (.999) = 82 dbSPL.

Conditions (Target Attention Filters) Conditions 1 and 2 were brightness-only and loudness-only attention condition judged with feedback whether the mean brightness (loudness) of the disks in the stimulus stream was higher versus lower than the usual. Condition 3: judge whether the stimulus stream contained a greater number of correlated (brightest/loudest plus dimmest/quietest) or anticorrelated (brightest/quietest plus dimmest/loudest) pairings The 9 token types using the three amplitude levels (0, 1, 2) for the auditory and visual component of each token. The five target filter matrices show the positive and negative weighting of each token depending on the experimental condition.

Sample Trials Embed video here. Mention feedback after each trial.

Basic Model In any given trial of a particular condition, we assume that the subject responds "1" (vs "-1" otherwise) if where k is the position of each token in the stimulus. Simple model for first 2 conds, binding params model for last 3 conds. In each condition, a probit model was used to measure the impact exerted on the observer’s judgments by each of the 9 types of pairings The criterion $Crit$ is selected by the participant to optimize performance. We assume that a standard normal random variable $Noise$ is added to the decision statistic.

Results: Attend to One Modality .55 .89 .52 .77 1.30 .65

Results: Attend to Both Modalities .39 .46 .38 .39 .65 .46 .44 .43

Principle Components Analysis Token weights scaled by sensitivity for each condition 3 dimensions of sensitivity given the conditions we tested Loudness (blue) Brightness (green) Correlatedness (red) Group PCA very similar to individual PCA. Cyan is the fourth dimension of the PCA to show that the other dimensions all wobbled around zero, not doing anything.

Elaborating the model: Binding Simultaneity Parameters Stimulus w(-2) w(2) w(1) w(-1) w(0) Explain subscripting.

Model with Binding Simultaneity Parameters In any given trial of a particular condition, we assume that the subject responds "1" (vs "-1" otherwise) if where where the A(k) is the auditory component and V(k) is the visual component of token k. f( A(k),V(k+ δ) ) is the token-type sensitivity for that token in that condition The Binding Simultaneity Parameters sum = 1 Markov Chain Monte Carlo simulation used to estimate the joint posterior density of the vector comprising the parameters used to fit the data

Results: Temporal Resolution

Temporal Resolution of Model Simulation Participant 1 Participant 2 Analogous Fujisaki experiment on our display revealed their result of 2-3Hz limit. This model was on data that operated on stimuli with low amplitudes instead of blank/silent. Our model simulation on phase judgment stimuli reveals higher binding temporal resolution than the Fujisaki & Nishida 2010 result (orange box).

Conclusions Observers can achieve a range of different audiovisual attention filters. There were 3 dimensions of attention sensitivity given our tasks. Dynamic random stimuli allowed multimodal binding at higher temporal acuity (3.8-5.25 Hz) than phase judgment stimuli (2-3 Hz) Suggests that auditory and visual information from randomly varying stimuli can be bound more precisely in time than strictly oscillating stimuli Brightness, loudness strongest PCs.

Acknowledgments Thanks for listening! The Chubb-Wright Lab at UC Irvine This work was funded by NSF Award BCS-0843897 Group photo of labmates? Thanks for listening!

Supplemental slides