Advance fMRI analysis: MVPA Sara Fabbri Brain and Mind Institute Western University http://www.fmri4newbies.com/ Advance fMRI analysis: MVPA

Overview Why MVPA? How MVPA “Mind-reading” MVPA classifier MVPA correlation Representational similarity approach “Mind-reading”

Response of a neuron from the human medial temporal lobe Picture number Picture Raster plots for various trials (from top to bottom) Firing rate onset offset Quiroga et al., 2005, Nature

Response of a neuron from the human medial temporal lobe Quiroga et al., 2005, Nature

Response of a neuron from the human medial temporal lobe Selective response to Jennifer Aniston pictures Neuron 1 “likes” Jennifer Aniston Quiroga et al., 2005, Nature

Firing Rate Firing Rate Firing Rate Neuron 1 “likes” Jennifer Aniston Julia Roberts Neuron 3 “likes” Brad Pitt Firing Rate Firing Rate Firing Rate

fMRI spatial resolution: 1 voxel high activity fMRI spatial resolution: 1 voxel 3 mm Fusiform Face Area (FFA) 3 mm 3 mm 3 mm low activity 3 mm A voxel might contain millions of neurons, so the fMRI signal represents the population activity

Firing Rate Firing Rate Firing Rate Activation Neuron 1 “likes” Jennifer Aniston Neuron 2 “likes” Julia Roberts Neuron 3 “likes” Brad Pitt Even though there are neurons tuned to each actor, the population as a whole shows no preference Firing Rate Firing Rate Firing Rate Activation

Limitation of Subtraction Logic Are there neurons that prefer Jennifer to Julia in the voxel? > Subtraction Activation Activation No preference

Two Techniques with “Subvoxel Resolution” “subvoxel resolution” = the ability to investigate coding in neuronal populations smaller than the voxel size being sampled Multi-Voxel Pattern Analysis (MVPA or decoding or “mind reading”) fMR Adaptation (or repetition suppression or priming)

Multi-Voxel Pattern Analyses

fMRI spatial resolution: 1 voxel high activity fMRI spatial resolution: 1 voxel 3 mm low activity 3 mm

Region Of Interest (ROI): group of voxels high activity 3 mm 3 mm low activity 3 mm

Standard fMRI Analysis FACES HOUSES trial 1 trial 1 trial 2 trial 2 trial 3 trial 3 Average Summed Activation

4 mm Full-width-Half-Maximum (FWHM) Spatial Smoothing 4 mm Full-width-Half-Maximum (FWHM) 7 mm FWHM 10 mm FWHM No smoothing Application of a Gaussian low-pass spatial filter to fMRI data resulting in spreading the activation across adjacent voxels Why? increase signal-to-noise facilitate intersubject averaging

Effect of Spatial Smoothing: reduce spatial resolution

Why Multi-Voxel Pattern Analysis (MVPA)? Patterns carry more information than the average across a pattern Average Pattern Standard fMRI Analysis MVPA fMRI Analysis

We have seen that the pattern of activity might be informative but it is lost in the standard fMRI analysis Questions?

How Multi-Voxel Pattern Analysis? MVPA classifier MVPA correlation Representational similarity approach

1. MVPA CLASSIFIER

FACES HOUSES trial 1 trial 1 Training Trials trial 2 trial 2 trial 3 … … Can an algorithm correctly “guess” trial identity better than chance (50%)? Test Trials (not in training set)

Voxel 1 Voxel 2 Activity in Voxel 1 Activity in Voxel 2 Faces Houses Each dot is one measurement (trial) from one condition (red circles) or the other (green circles) Activity in Voxel 2 Faces Houses

Training set Test set Activity in Voxel 1 Activity in Voxel 2 Faces Houses Classifier

THIS STEP IS THE CORE OF THE ANALYSIS BECAUSE IT TESTS THE ABILITY OF THE CLASSIFIER TO GENERALIZE KNOWLEDGE TO NEW DATA Test set Activity in Voxel 1 Activity in Voxel 2 Faces Correct 6 Classifier Accuracy = = = 75 % Houses 8 Incorrect Classifier

Faces Houses In a, the classifier can operate on single voxels because the response distributions are separable within individual voxels. Cox and Savoy (2003)

Also standard fMRI analysis can detect this difference Faces Houses Also standard fMRI analysis can detect this difference Cox and Savoy (2003)

Faces Houses In b, the classifier cannot operate on single voxels because the response distributions are overlapping within individual voxels. Cox and Savoy (2003)

Standard fMRI analysis cannot detect this difference Faces Houses Standard fMRI analysis cannot detect this difference Cox and Savoy (2003)

In c, the non-linear classifier draws decision boundaries other than straight lines Cox and Savoy (2003)

Standard fMRI analysis cannot detect this difference Cox and Savoy (2003)

Support Vector Machine (SVM) SVM is a linear decision boundary that discriminates between two sets of points with the constrain of a greater distance from the closest points on both sides. The response patterns closest to the decision boundary (yellow circles) that defined the margins are called “support vectors”. Mur et al., 2009 Mur et al., 2009

Choices to make Feature selection to remove noisy and/or uninformative voxels before classification: Limit the analysis to specific anatomical regions Compute univariate (voxel-wise) statistics Cross-validation options: Leave-one-trial out: for each condition train the classifier with all but one trial and use the left out subset as test data. Repeat this procedure until each subset has been used as test data once. Leave-one-run out: train the classifier with all but one runs and test on the excluded run. Repeat until all runs are used as test data once. Performance evaluation options: T-test versus 50% chance decoding Permutation test: train and test the classifier with labels (e.g. faces and houses) randomly assigned to the data. Compare classifier performance on real data with performance on randomized data.

Example of MVPA classifier approach: decoding future actions Gallivan et al., 2011

Conditions Gallivan et al., 2011

Delayed-paradigm Gallivan et al., 2011

PREDICT FUTURE EFFECTOR PREDICT THE UPCOMING REACH OR SACCADE Gallivan et al., 2011

PREDICT FUTURE DIRECTION PREDICT THE UPCOMING MOVEMENT DIRECTION Gallivan et al., 2011

PREDICT FUTURE ACTION! Gallivan et al., 2011

What else can we examine? So far we’ve only examined the question: Can we discriminate Condition 1 vs. Condition 2 from brain activation patterns? New question: Are different stimuli similarly represented in certain ROIs? Answer: Cross-decoding: train to discriminate between Condition 1 and 2 and test classifier accuracy with Condition 3 and 4 Fraser Smith & Jason Gallivan

Cross-trial-type decoding Training set Activity in Voxel 1 Activity in Voxel 2 Faces Houses Classifier Fraser Smith & Jason Gallivan

Cross-trial-type decoding Test set Activity in Voxel 1 Activity in Voxel 2 Animals Tools Classifier Fraser Smith & Jason Gallivan

Similar spatial encoding across the eye and the hand Train to discriminate movement directions for one effector (HandL vs HandR) and test it on the other effector (EyeL vs EyeR) Gallivan et al., 2011

Effector specificity across different movement directions Train to discriminate the effector for one movement directions (EyeL vs HandL) and test it on the other direction (EyeR vs HandR) Gallivan et al., 2011

Effector specificity across different movement direction Gallivan et al., 2011

We have seen how MVPA classifier can be used to distinguish between different stimulus categories based on the pattern of activity Questions?

2. CORRELATION APPRAOCH

MVPA correlation approach Faces Houses trial 1 trial 1 trial 1 trial 2 trial 2 trial 2 trial 3 trial 3 trial 3 trial 3 Average Summed Activation

MVPA correlation approach Faces Houses trial 1 trial 1 trial 1 trial 1 trial 2 trial 2 trial 2 trial 2 trial 3 trial 3 trial 3 trial 3 trial 3 Average Summed Activation The same category evokes similar patterns of activity across trials

MVPA correlation approach Faces Houses trial 1 trial 1 trial 1 trial 2 trial 2 trial 2 trial 3 trial 3 trial 3 trial 3 Average Summed Activation Similarity Within the same category

MVPA correlation approach Faces Houses trial 1 trial 1 trial 1 trial 1 trial 2 trial 2 trial 2 trial 2 trial 3 trial 3 trial 3 trial 3 Average Summed Activation Similarity Between different categories

The brain area contains distinct information about faces and houses Within-category similarity Between-category similarity > The brain area contains distinct information about faces and houses

Category-specificity of patterns of response in the ventral temporal cortex Haxby et al., 2001, Science

Within-category similarity Category-specificity of patterns of response in the ventral temporal cortex SIMILARITY MATRIX ODD RUNS high EVEN RUNS similarity low Within-category similarity Haxby et al., 2001, Science

Between-category similarity Category-specificity of patterns of response in the ventral temporal cortex SIMILARITY MATRIX ODD RUNS high EVEN RUNS similarity low Between-category similarity Haxby et al., 2001, Science

fMRI activity high similarity low Bracci et al., 2011

Similar representation for tools and hand not identified from the standard univariate analysis fMRI activity high similarity low Bracci et al., 2011

Searchlight analysis We have seen how to search for patterns of information within a region of interest, but what if we want to know “Where in the brain does the activity pattern contain information about the experimental condition?” Scanned the brain with a spherical multivariate “searchlight” centered on each voxel in turn (optimal searchlight has radius of 4 mm, that contains 33 2-mm-isovoxel voxels) Compute multivariate effect statistic at each location Kriegeskorte et al., 2006

We have seen how MVPA correlation can measure similarities and differences between patterns of activity across conditions Questions?

3. REPRESENTATIONAL SIMILARITY APPROACH (RSA)

PRE-DEFINED CATEGORIES CATEGORY A CATEGORY B Candies in different boxes are different because they have different prices

Which dimension to choose for the classification? HOW ARE THESE OBJECTS CLASSIFIED IN THE BRAIN (BY PRIC€, COLOR, SHAPE, FLAVOUR OR OTHER DIMENSIONS)?

Representational similarity approach (RSA) Differently from the MVPA correlation, RSA does not separate stimuli into a-priori categories MVPA correlation RSA high low similarity ODD RUNS EVEN RUNS Kriegeskorte et al (2008)

No class boundaries! C1 high . . CONDITIONS TRIALS similarity low C96 Fraser Smith & Jason Gallivan

TEST VARIOUS PREDICTIONS high low similarity Kriegeskorte et al (2008)

TEST VARIOUS PREDICTIONS Which prediction matrix is more similar to the real data? high low similarity REAL DATA Kriegeskorte et al (2008)

Interspecies comparisons Man + monkey?? similarity high low Kriegeskorte et al., 2008 Fraser Smith & Jason Gallivan

Activation- vs. information-bases analysis Activation-based (standard fMRI analysis): regions more strongly active during face than house perception Information-based (searchlight MVPA analysis): regions whose activity pattern distinguished the two categories 35 % of voxels are marked only in the information-based map: category information is lost when data are smoothed Kriegeskorte, Goebel & Bandettini, 2006

Activation- vs. information-based analysis Mur et al., 2009, Social Cognitive and Affective Neuroscience

Limitations of MVPA Good classification indicates presence of information (no necessarily neuronal selectivity) (Logothetis, 2008). E.g. successful face decoding in primary visual cortex Pattern-classifier analysis requires many decisions that affect the results (see Misaki et al., 2010)

We have seen how RSA can be used for exploratory analysis with complex stimulus set To summarize, the three MVPA methods are powerful especially to study distributed representations Questions?

“Mind-Reading”: Reconstructing new stimuli from brain activity

Reconstruct new images The model is trained to classify the pattern on brain activity in primary visual cortex during observation of several hundred random images The model can accurately reconstruct arbitrary images, including geometric shapes on a single 2 sec volume without any prior information about the image Miyawaki et al., 2008

Reconstruct new movies 3 participants watched hours of movie trailers during fMRI Reconstruction of new movies from brain activity during similar clips from YouTube Presented movies Reconstructed movies http://www.youtube.com/watch?v=nsjDnYxJ0bo (Nishimoto et al., 2011

Lie detector Non-linear classifier applied to fMRI data to discriminate spatial patterns of activity associated to lie and truth in 22 individual participants. 88% accuracy to detect lies in participants not included in the training (Davatzikos et al., 2005)

Lie detector Non-linear classifier applied to fMRI data to discriminate spatial patterns of activity associated to lie and truth in 22 individual participants. 88% accuracy to detect lies in participants not included in the training The real world is more complex!

Reconstruct dreams Measure brain activity while 3 participants were asleep and ask them to describe their dream when awake Comparison between brain activity during sleep and vision of pictures of categories frequently dreamt Activity in higher order visual areas (i.e. FFA) could successfully (accuracy of 75-80%) decode the dream contents 9 seconds before waking the participant! Abstract SfN 2012: Dreaming is a subjective experience during sleep, often accompanied by visual contents, whose neural basis remains unknown. Previous dream research attempted to link physiological states with dreaming, but did not demonstrate how the specific contents of visual experiences during dreaming are represented in the brain activity patterns. The recent advent of neural decoding has allowed for the decoding of various contents of visual experience from brain activity patterns. The technique can thus be used to examine the neural representation of dreams by testing whether neural decoders can predict dream contents from brain activity patterns. Here we performed decoding analyses on semantically labeled human fMRI signals measured from three male subjects during dreaming. To collect dream data efficiently, we measured fMRI signals and collected reports about subjective experiences during hypnagogic periods. We developed a multiple-awakening procedure, in which subjects were awakened when a specific EEG pattern was observed, were asked to freely describe their visual experiences just before awakening, and were then asked to sleep again. We repeated this procedure until we collected over 200 reports in a total of 30 - 45 hours of experiment time for each subject. Multiple “synsets,” synonym sets defined in the English “WordNet” lexical database, that correspond to words describing reported visual contents were used to label averaged fMRI volumes during a 9 s period before each awakening. We first performed pairwise classification analyses for all pairs of synsets using fMRI signals in the early (V1-V3) and the higher (around LOC, FFA, and PPA) visual cortices during dreaming (dream-trained decoder). The decoding performance showed a distribution that was significantly higher than chance level in the higher visual areas. We next examined whether “stimulus-trained decoders” that were trained with fMRI signals evoked by natural image viewing could decode the dream contents. Results showed that the stimulus-trained decoders successfully predicted the dream contents more accurately in the higher visual cortex than in the early visual cortex. These results demonstrate that fMRI signals in the visual cortex, especially in the higher visual areas, represent specific visual contents of dreams, allowing for the prediction of dream contents. Furthermore, it supports the hypothesis that dreaming and perception may share neural representations in the higher visual areas. Horikawa, Tamaki, Miyawaki and Kamitani, Society for Neuroscience 2012

Thousand of categories in the real world Which of the thousand of categories are represented similarly in the brain and which aren’t? Measure brain activity from 5 subjects while watching movies See example movies: http://www.youtube.com/watch?feature=player_embedded&v=u9nMfaWqkVE

Shared Semantic Space from brain activity during observation of movies Similar colors for categories similarly represented in the brain Huth et al., 2012

Shared Semantic Space from brain activity during observation of movies Similar colors for categories similarly represented in the brain People and communication verbs are represented similarly Huth et al., 2012

Continuous Semantic Space across the surface Each voxel is colored accordingly to which part of the semantic space is selective for http://gallantlab.org/semanticmovies/

Continuous Semantic Space across the surface Click on each voxel to see which categories it represents FUSIFORM FACE AREA http://gallantlab.org/semanticmovies/

Continuous Semantic Space across the surface Click on each category to see how it is represented in the brain FACE http://gallantlab.org/semanticmovies/

The possibility to decode brain activity has great potentials for brain-computer applications http://www.youtube.com/watch?v=TJJPbpHoPWo Let’s watch a video! http://www.youtube.com/watch?v=TJJPbpHoPWo

The end fabbri.sara@gmail.com