Multi-voxel Pattern Analysis (MVPA) and “Mind Reading” By: James Melrose

Basic Visual Pathways

function Magnetic Resonance Imaging (fMRI) Based on the idea that oxygenated hemoglobin has slightly different magnetic properties than nonoxygenated hemoglobin and that areas of the brain being used will require this oxygen

fMRI cont. Hydrogen protons within the brain align with the huge magnet in the scanner, radio waves are then aimed towards the head causing the rotation of the Hydrogen to misalign, the atoms with oxygen containing hemoglobin realign at a different rate than atoms without oxygen, the realignment sends out a signal that can then be detected by the detector coil.

MVPA as alternative to traditional fMRI Instead of focusing on single voxels (the fMRI version of a pixel), MVPA uses pattern- classification algorithms applied to multiple voxels to decode the patterns of activity Due to its increased sensitivity, it makes it easier to get statistically significant brain activation Also reduces the problem of noise when looking at individual voxels

MVPA 4 Steps: Feature selection- decide which voxels you want to include in analysis Pattern assembly- “involves sorting the data into discrete brain patterns corresponding to the pattern of activity across the selected voxels at a particular time in the experiment” Classifier training- feed these patterns into multivariate pattern classification algorithm that maps between voxel patterns and experimental conditions Generalization testing- test to see if the trained classifier algorithm can determine the correct experimental condition associated with activation pattern

How to develop MVPA paradigm

Kamitani and Tong Perception of edge orientation can be detected from fMRI using MVPA and a 3x3x3 matrix of voxels in early visual areas (V1 & V2)

Kamitani and Tong cont. The detection of edge orientation can be extended when subjects are told to focus on one of two line orientation in a grid

Haxby et al. Had subjects look at pictures and line drawings of faces, cats, man-made objects, and control nonsense objects. The pattern of response in object selective ventral temporal cortex correctly identified the category being viewed in 96% of comparisons.

Haxby et al., cont.

Ability of both photographs and line drawings to elicit the same accuracy of prediction, the representation being detected must be of a higher level representation of the image rather than simple feature detection

What else has MVPA done? Mind-reading has been accomplished in the domains of: Direction of dots Picture vs. sentence Ambiguous vs. non-ambiguous sentences Category of viewed word Which of 3 categories thinking of Lying

Where can this go? The implications of this research could potentially lead to researchers being able to determine what a person is looking at or thinking about entirely based on their pattern of brain activation Perhaps even one day having subjects fall asleep in the scanner and be able to tell what the subject is dreaming of

References Haxby, J.V. et al. (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, Kamitani, Y. and Tong, F. (2005) Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8, Norman, K.A.; Polyn, S.M.; Detre, G.J.; Haxby, J.V. (2006) Beyond mind reading: multi-voxel pattern analysis of fMRI data. Trends in Cognitive Sciences. 10 (9),