Sonification of fMRI Data Nik Sawe Music 220C

Overview PhD studies assess decision-making on environmental issues through neuroimaging Neural activation suggests underlying physiological bases for framing effects, heuristics, affect (emotion) and their impact on decision-making

How We Image the Brain Functional MRI allows us to take realtime pictures of the brain’s response to stimuli. Using headsets and hand input devices, can present subjects with a wide range of tasks.

The BOLD Signal BOLD: Blood Oxygenation Level-Dependent fMRI evaluates brain activity indirectly, by measuring changes in the local amount of oxygenated blood Complex regressions account for fluctuations due to heart rate, breathing, etc. Validity confirmed through optogenetics

Motivations for Sonification Can hear patterns of activation that would be less obvious through visualization of time courses

Motivations for Sonification May be able to hear “conversations” between different brain regions that would be less obvious through traditional neuroimaging analyses Intuitive level of interpretation that may provide clues for further analytic techniques

Limitations of fMRI Poor temporal resolution – One pass through each brain region every 1-2 seconds (most often 2)

Limitations of fMRI Poor temporal resolution – One pass through each brain region every 1-2 seconds (most often 2) For most study designs, need many repeated trials in one person to get an accurate read

Translatable fMRI Outputs

Sonification Methodology Built in R from a simple initial formula – Pitch: 128 * [(X i – X l )/(X h -X l )] – Velocity: 128*P i X i : signal at timepoint i X h : maximum signal X l : minimum signal P i : a given network's proportional contribution to the total signal strength of all sampled networks at timepoint i

Sonification Methodology Use these new values as downstream MIDI values, convert to MIDI file via Java First trial: utilized data from one subject in my first study (environmental philanthropy to save parks threatened with potentially destructive land use development) Used 3 networks: attentional, visual, default mode network

Visual Cortex Quartet Final project: Sampled from the visual cortex as subject undergoes retinotopy

Sonification Methodology Program had several stages: Scale converter: created array of MIDI values based on desired scale Instrument filter: selected valid (in scale range) notes for a given instrument Signal to MIDI converter: Gated signals below a threshold value (5%) and did not play them

Sonification Methodology Velocity based upon relative prominence of the voxel signal given other voxels’ activity Duration based on arbitrary equation of: – ((128-note value)+velocity)/20

The Next Step Scan whole brain while watching a silent film May obtain complementary EEG data Will have PCA networks to work with, as well as a wealth of regions Signals do not all have to be pitch modulation

Mapping Ideas Activity in the attendant PCA network helps define duration and velocity for each region, based on its relative contribution Talairach (spatial) coordinates define surround sound mapping

Mapping: Anterior Insula Handles “negative arousal” / response to physiologically as well as morally aversive stimuli Control how discordant the note selection is in other regions

Nucleus Accumbens Handles “positive arousal”/reward/approach behavior Control weighting towards major scales May be able to make a balancing equation of AI vs Nacc

Mapping: Amygdala Fear/apprehension/anxiety region Control tempo, accelerating at tense moments Control percussive elements Trigger clusters

Mapping: Fusiform Gyrus Recognizes faces: triggering of voice samples?

Parahippocampal Gyrus Spatial/landscape encoding Spatial manipulation of samples/Doppler?

Incorporation of EEG Since temporal resolution is only 2 sec passes, would be good to have variation that decides interleaving of notes This interpolation can be decided by activity in relevant EEG signals

Thanks!