1. Advance fMRI analysis: MVPA
Sara Fabbri
Brain and Mind Institute
Western University
Advance fMRI analysis: MVPA

2. Overview Why MVPA? How MVPA “Mind-reading” MVPA classifier
MVPA correlation
Representational similarity approach
“Mind-reading”

4. Response of a neuron from the human medial temporal lobe
Picture number
Picture
Raster plots for various trials (from top to bottom)
Firing rate
onset
offset
Quiroga et al., 2005, Nature

5. Response of a neuron from the human medial temporal lobe
Quiroga et al., 2005, Nature

6. Response of a neuron from the human medial temporal lobe
Selective response to Jennifer Aniston pictures
Neuron 1
“likes”
Jennifer Aniston
Quiroga et al., 2005, Nature

7. Firing Rate Firing Rate Firing Rate Neuron 1 “likes” Jennifer Aniston
Julia Roberts
Neuron 3
“likes”
Brad Pitt
Firing Rate
Firing Rate
Firing Rate

8. fMRI spatial resolution: 1 voxel
high activity
fMRI spatial resolution: 1 voxel
3 mm
Fusiform Face Area (FFA)
3 mm
3 mm
3 mm
low activity
3 mm
A voxel might contain millions of neurons, so the fMRI signal represents the population activity

9. Firing Rate Firing Rate Firing Rate Activation
Neuron 1
“likes”
Jennifer Aniston
Neuron 2
“likes”
Julia Roberts
Neuron 3
“likes”
Brad Pitt
Even though there are neurons tuned to each actor, the population as a whole shows no preference
Firing Rate
Firing Rate
Firing Rate
Activation

10. Limitation of Subtraction Logic
Are there neurons that prefer Jennifer to Julia in the voxel?
>
Subtraction
Activation
Activation
No preference

11. Two Techniques with “Subvoxel Resolution”
“subvoxel resolution” = the ability to investigate coding in neuronal populations smaller than the voxel size being sampled
Multi-Voxel Pattern Analysis (MVPA or decoding or “mind reading”)
fMR Adaptation (or repetition suppression or priming)

12. Multi-Voxel Pattern Analyses

13. fMRI spatial resolution: 1 voxel
high activity
fMRI spatial resolution: 1 voxel
3 mm
low activity
3 mm

14. Region Of Interest (ROI): group of voxels
high activity
3 mm
3 mm
low activity
3 mm

15. Standard fMRI Analysis
FACES
HOUSES
trial 1
trial 1
trial 2
trial 2
trial 3
trial 3
Average Summed
Activation

16. 4 mm Full-width-Half-Maximum (FWHM)
Spatial Smoothing
4 mm Full-width-Half-Maximum (FWHM)
7 mm FWHM
10 mm FWHM
No smoothing
Application of a Gaussian low-pass spatial filter to fMRI data resulting in spreading the activation across adjacent voxels
Why?
increase signal-to-noise
facilitate intersubject averaging

17. Effect of Spatial Smoothing: reduce spatial resolution

18. Why Multi-Voxel Pattern Analysis (MVPA)?
Patterns carry more information than the average across a pattern
Average
Pattern
Standard fMRI Analysis
MVPA fMRI Analysis

19. We have seen that the pattern of activity might be informative but it is lost in the standard fMRI analysis
Questions?

20. How Multi-Voxel Pattern Analysis?
MVPA classifier
MVPA correlation
Representational similarity approach

21. 1. MVPA CLASSIFIER

22. FACES HOUSES trial 1 trial 1 Training Trials trial 2 trial 2 trial 3… …
Can an algorithm correctly “guess” trial identity better than chance (50%)?
Test Trials
(not in training set)

23. Voxel 1 Voxel 2 Activity in Voxel 1 Activity in Voxel 2 Faces Houses
Each dot is one measurement (trial) from one condition (red circles) or the other (green circles)
Activity in Voxel 2
Faces
Houses

24. Training set
Test set
Activity in Voxel 1
Activity in Voxel 2
Faces
Houses
Classifier

25. THIS STEP IS THE CORE OF THE ANALYSIS BECAUSE IT TESTS THE ABILITY OF THE CLASSIFIER TO GENERALIZE KNOWLEDGE TO NEW DATA
Test set
Activity in Voxel 1
Activity in Voxel 2
Faces
Correct 6
Classifier Accuracy = = =
75 %
Houses 8
Incorrect
Classifier

26. Faces
Houses
In a, the classifier can operate on single voxels because the response distributions are separable within individual voxels.
Cox and Savoy (2003)

27. Also standard fMRI analysis can detect this difference
Faces
Houses
Also standard fMRI analysis can detect this difference
Cox and Savoy (2003)

28. Faces
Houses
In b, the classifier cannot operate on single voxels because the response distributions are overlapping within individual voxels.
Cox and Savoy (2003)

29. Standard fMRI analysis cannot detect this difference
Faces
Houses
Standard fMRI analysis cannot detect this difference
Cox and Savoy (2003)

30. In c, the non-linear classifier draws decision boundaries other than straight lines
Cox and Savoy (2003)

31. Standard fMRI analysis cannot detect this difference
Cox and Savoy (2003)

32. Support Vector Machine (SVM)
SVM is a linear decision boundary that discriminates between two sets of points with the constrain of a greater distance from the closest points on both sides.
The response patterns closest to the decision boundary (yellow circles) that defined the margins are called “support vectors”.
Mur et al., 2009
Mur et al., 2009

33. Choices to make
Feature selection to remove noisy and/or uninformative voxels before classification:
Limit the analysis to specific anatomical regions
Compute univariate (voxel-wise) statistics
Cross-validation options:
Leave-one-trial out: for each condition train the classifier with all but one trial and use the left out subset as test data. Repeat this procedure until each subset has been used as test data once.
Leave-one-run out: train the classifier with all but one runs and test on the excluded run. Repeat until all runs are used as test data once.
Performance evaluation options:
T-test versus 50% chance decoding
Permutation test: train and test the classifier with labels (e.g. faces and houses) randomly assigned to the data. Compare classifier performance on real data with performance on randomized data.

34. Example of MVPA classifier approach: decoding future actions
Gallivan et al., 2011

35. Conditions
Gallivan et al., 2011

36. Delayed-paradigm
Gallivan et al., 2011

37. PREDICT FUTURE EFFECTOR
PREDICT THE UPCOMING REACH OR SACCADE
Gallivan et al., 2011

38. PREDICT FUTURE DIRECTION
PREDICT THE UPCOMING MOVEMENT DIRECTION
Gallivan et al., 2011

39. PREDICT FUTURE ACTION!
Gallivan et al., 2011

40. What else can we examine?
So far we’ve only examined the question: Can we discriminate Condition 1 vs. Condition 2 from brain activation patterns? New question: Are different stimuli similarly represented in certain ROIs? Answer: Cross-decoding: train to discriminate between Condition 1 and 2 and test classifier accuracy with Condition 3 and 4
Fraser Smith & Jason Gallivan

41. Cross-trial-type decoding
Training set
Activity in Voxel 1
Activity in Voxel 2
Faces
Houses
Classifier
Fraser Smith & Jason Gallivan

42. Cross-trial-type decoding
Test set
Activity in Voxel 1
Activity in Voxel 2
Animals
Tools
Classifier
Fraser Smith & Jason Gallivan

43. Similar spatial encoding across the eye and the hand
Train to discriminate movement directions for one effector (HandL vs HandR) and test it on the other effector (EyeL vs EyeR)
Gallivan et al., 2011

44. Effector specificity across different movement directions
Train to discriminate the effector for one movement directions (EyeL vs HandL) and test it on the other direction (EyeR vs HandR)
Gallivan et al., 2011

45. Effector specificity across different movement direction
Gallivan et al., 2011

46. We have seen how MVPA classifier can be used to distinguish between different stimulus categories based on the pattern of activity
Questions?

47. 2. CORRELATION APPRAOCH

48. MVPA correlation approach
Faces
Houses
trial 1
trial 1
trial 1
trial 2
trial 2
trial 2
trial 3
trial 3
trial 3
trial 3
Average Summed
Activation

49. MVPA correlation approach
Faces
Houses
trial 1
trial 1
trial 1
trial 1
trial 2
trial 2
trial 2
trial 2
trial 3
trial 3
trial 3
trial 3
trial 3
Average Summed
Activation
The same category evokes similar patterns of activity across trials

50. MVPA correlation approach
Faces
Houses
trial 1
trial 1
trial 1
trial 2
trial 2
trial 2
trial 3
trial 3
trial 3
trial 3
Average Summed
Activation
Similarity Within the same category

51. MVPA correlation approach
Faces
Houses
trial 1
trial 1
trial 1
trial 1
trial 2
trial 2
trial 2
trial 2
trial 3
trial 3
trial 3
trial 3
Average Summed
Activation
Similarity Between different categories

52. The brain area contains distinct information about faces and houses
Within-category similarity
Between-category similarity
>
The brain area contains distinct information about faces and houses

53. Category-specificity of patterns of response in the ventral temporal cortex
Haxby et al., 2001, Science

54. Within-category similarity
Category-specificity of patterns of response in the ventral temporal cortex
SIMILARITY MATRIX
ODD RUNS
high
EVEN RUNS
similarity
low
Within-category similarity
Haxby et al., 2001, Science

55. Between-category similarity
Category-specificity of patterns of response in the ventral temporal cortex
SIMILARITY MATRIX
ODD RUNS
high
EVEN RUNS
similarity
low
Between-category similarity
Haxby et al., 2001, Science

56. fMRI activity
high
similarity
low
Bracci et al., 2011

57. Similar representation for tools and hand not identified from the standard univariate analysis
fMRI activity
high
similarity
low
Bracci et al., 2011

58. Searchlight analysis
We have seen how to search for patterns of information within a region of interest, but what if we want to know
“Where in the brain does the activity pattern contain information about the experimental condition?”
Scanned the brain with a spherical multivariate “searchlight” centered on each voxel in turn (optimal searchlight has radius of 4 mm, that contains 33 2-mm-isovoxel voxels)
Compute multivariate effect statistic at each location
Kriegeskorte et al., 2006

59. We have seen how MVPA correlation can measure similarities and differences between patterns of activity across conditions
Questions?

60. 3. REPRESENTATIONAL SIMILARITY APPROACH (RSA)

61. PRE-DEFINED CATEGORIES
CATEGORY A
CATEGORY B
Candies in different boxes are different because they have different prices

62. Which dimension to choose for the classification?
HOW ARE THESE OBJECTS CLASSIFIED IN THE BRAIN
(BY PRIC€, COLOR, SHAPE, FLAVOUR OR OTHER DIMENSIONS)?

63. Representational similarity approach (RSA)
Differently from the MVPA correlation, RSA does not separate stimuli into a-priori categories
MVPA correlation
RSA
high
low
similarity
ODD RUNS
EVEN RUNS
Kriegeskorte et al (2008)

64. No class boundaries! C1 high . . CONDITIONS TRIALS similarity low C96
Fraser Smith & Jason Gallivan

65. TEST VARIOUS PREDICTIONS
high
low
similarity
Kriegeskorte et al (2008)

66. TEST VARIOUS PREDICTIONS
Which prediction matrix is more similar to the real data?
high
low
similarity
REAL
DATA
Kriegeskorte et al (2008)

67. Interspecies comparisons
Man + monkey??
similarity
high
low
Kriegeskorte et al., 2008
Fraser Smith & Jason Gallivan

68. Activation- vs. information-bases analysis
Activation-based (standard fMRI analysis):
regions more strongly active during face than house perception
Information-based (searchlight MVPA analysis):
regions whose activity pattern distinguished the two categories
35 % of voxels are marked only in the information-based map: category information is lost when data are smoothed
Kriegeskorte, Goebel & Bandettini, 2006

69. Activation- vs. information-based analysis
Mur et al., 2009, Social Cognitive and Affective Neuroscience

70. Limitations of MVPA
Good classification indicates presence of information (no necessarily neuronal selectivity) (Logothetis, 2008). E.g. successful face decoding in primary visual cortex
Pattern-classifier analysis requires many decisions that affect the results (see Misaki et al., 2010)

71. We have seen how RSA can be used for exploratory analysis with complex stimulus set
To summarize, the three MVPA methods are powerful especially to study distributed representations
Questions?

72. “Mind-Reading”: Reconstructing new stimuli from brain activity

73. Reconstruct new images
The model is trained to classify the pattern on brain activity in primary visual cortex during observation of several hundred random images
The model can accurately reconstruct arbitrary images, including geometric shapes on a single 2 sec volume without any prior information about the image
Miyawaki et al., 2008

74. Reconstruct new movies
3 participants watched hours of movie trailers during fMRI
Reconstruction of new movies from brain activity during similar clips from YouTube
Presented movies
Reconstructed movies
(Nishimoto et al., 2011

75. Lie detector
Non-linear classifier applied to fMRI data to discriminate spatial patterns of activity associated to lie and truth in 22 individual participants.
88% accuracy to detect lies in participants not included in the training
(Davatzikos et al., 2005)

76. Lie detector
Non-linear classifier applied to fMRI data to discriminate spatial patterns of activity associated to lie and truth in 22 individual participants.
88% accuracy to detect lies in participants not included in the training
The real world is more complex!

77. Reconstruct dreams
Measure brain activity while 3 participants were asleep and ask them to describe their dream when awake
Comparison between brain activity during sleep and vision of pictures of categories frequently dreamt
Activity in higher order visual areas (i.e. FFA) could successfully (accuracy of 75-80%) decode the dream contents 9 seconds before waking the participant!
Abstract SfN 2012: Dreaming is a subjective experience during sleep, often accompanied by visual contents, whose neural basis remains unknown. Previous dream research attempted to link physiological states with dreaming, but did not demonstrate how the specific contents of visual experiences during dreaming are represented in the brain activity patterns. The recent advent of neural decoding has allowed for the decoding of various contents of visual experience from brain activity patterns. The technique can thus be used to examine the neural representation of dreams by testing whether neural decoders can predict dream contents from brain activity patterns. Here we performed decoding analyses on semantically labeled human fMRI signals measured from three male subjects during dreaming. To collect dream data efficiently, we measured fMRI signals and collected reports about subjective experiences during hypnagogic periods. We developed a multiple-awakening procedure, in which subjects were awakened when a specific EEG pattern was observed, were asked to freely describe their visual experiences just before awakening, and were then asked to sleep again. We repeated this procedure until we collected over 200 reports in a total of hours of experiment time for each subject. Multiple “synsets,” synonym sets defined in the English “WordNet” lexical database, that correspond to words describing reported visual contents were used to label averaged fMRI volumes during a 9 s period before each awakening. We first performed pairwise classification analyses for all pairs of synsets using fMRI signals in the early (V1-V3) and the higher (around LOC, FFA, and PPA) visual cortices during dreaming (dream-trained decoder). The decoding performance showed a distribution that was significantly higher than chance level in the higher visual areas. We next examined whether “stimulus-trained decoders” that were trained with fMRI signals evoked by natural image viewing could decode the dream contents. Results showed that the stimulus-trained decoders successfully predicted the dream contents more accurately in the higher visual cortex than in the early visual cortex. These results demonstrate that fMRI signals in the visual cortex, especially in the higher visual areas, represent specific visual contents of dreams, allowing for the prediction of dream contents. Furthermore, it supports the hypothesis that dreaming and perception may share neural representations in the higher visual areas.
Horikawa, Tamaki, Miyawaki and Kamitani, Society for Neuroscience 2012

78. Thousand of categories in the real world
Which of the thousand of categories are represented similarly in the brain and which aren’t?
Measure brain activity from 5 subjects while watching movies
See example movies:

79. Shared Semantic Space from brain activity during observation of movies
Similar colors for categories similarly represented in the brain
Huth et al., 2012

80. Shared Semantic Space from brain activity during observation of movies
Similar colors for categories similarly represented in the brain
People and communication verbs are represented similarly
Huth et al., 2012

81. Continuous Semantic Space across the surface
Each voxel is colored accordingly to which part of the semantic space is selective for

82. Continuous Semantic Space across the surface
Click on each voxel to see which categories it represents
FUSIFORM FACE AREA

83. Continuous Semantic Space across the surface
Click on each category to see how it is represented in the brain
FACE

84. The possibility to decode brain activity has great potentials for brain-computer applications
Let’s watch a video!

85. The end