More on neural similarity

Class project: the next steps I have replied to many, but not yet all, of your project proposals. I’ll have replied to all by end of this week. Sorry! What is known about this topic? Find a recent review article (e.g. using “review” term in PubMed, Trends in Cognitive Sciences, Current Opinion in Neurobiology) Find papers that cite this review (Google Scholar). See what Qs they are looking at. Try to identify an open question that hasn’t been looked at yet

Class project: the next steps Guiding principles: Would the imaging provide information over and above what we can get from behaviour? When people look the same from outside the head (behaviour), but are different inside (imaging) Maybe imaging can catch early signs of a process which won’t manifest itself in behaviour till later? Look at representations and mechanisms, not just what lit up Structure of representations, e.g. via neural similarity Try to relate neural activation to behaviour

Neural similarity: Does it actually matter? Some good criteria of whether something matters: Relation to behaviour Captures an underlying regularity across different people

Why do we remember some things but forget others? Encoding and retrieval Encoding hypothesis: Remembered items are better encoded Retrieval hypothesis: Remembered items are encoded the same, but the subsequent retrieval process grabs them more successfully

An early influential fMRI paper: Wagner et al., 1998 Wagner, A. D., Schacter, D. L., Rotte, M., Koutstaal, W., Maril, A., Dale, A. M., & Buckner, R. L. (1998). Building memories: remembering and forgetting of verbal experiences as predicted by brain activity. Science, 281(5380), 1188-1191.

Wagner et al. (1998) Greater activation at encoding time for subsequently remembered items. See also Brewer et al., same issue of Science. Pictures instead of words

Take-home message from Wagner et al. (1998) The subsequently remembered items are better encoded The better encoded items elicit more intense activation when they are presented

But what about neural pattern similarity, instead of activation intensity? Xue, G., Dong, Q., Chen, C., Lu, Z., Mumford, J. A., & Poldrack, R. A. (2010). Greater neural pattern similarity across repetitions is associated with better memory. Science, 330(6000), 97-101.

Xue et al., 2010 Test the “encoding variability hypothesis” Greater encoding variability leads to better subsequent memory Analogy: To become good at tennis, practice playing against lots of different partners, not just the same opponent over and over

Opposite hypothesis: Encoding similarity hypothesis To remember an item better, each presentation should induce a similar pattern of activation to the previous times Analogy: If you want to make a deep dent with a hammer, hit the same spot each time

The encoding similarity hypothesis in terms of neural similarity Each face is presented four times Hypothesised neural similarity of each presentation of subsequently remembered vs. forgotten faces

So, what did they find? Support for encoding similarity hypothesis: The subsequently remembered faces had greater neural similarity across presentations

Is this actually driven by neural pattern similarity? We already know from Wagner et al. (1998) that simply having greater activation intensity at encoding time gives better memory Maybe that’s driving the effect here, not greater neural pattern similarity Control: Calculate pattern similarity, after removing all the voxels that were more active for subsequently remembered items Result: Remembered items still show greater neural pattern similarity. So, it’s not just activation intensity

Wanted: Population-level regularities Say bridging spiel here

Wanted: Population-level regularities in the neural representation of information Aims: Find population-level regularities Challenges: How can we aggregate pattern- information across different people’s brains?

Locations of higher-level functional areas show variability across subjects From Georg Langs et al. (2011), Inf Process Med Imaging. “Learning an atlas of a cognitive process in its functional geometry”

Different people’s brains: alike at coarse-scale, different at fine-scale I can align my hand to your hand, and the fingers will match up But the fingerprints won’t match up

Just like literal fingerprints, neural fingerprints seem to be subject-unique Shinkareva, Mitchell and colleagues (PLoS ONE, 2008): Attempted both within- and across-subject decoding Found that “a critical diagnostic portion of the neural representation of the categories and exemplars is still idiosyncratic to individual participants”

Hyper-alignment A common, high-dimensional model of the representational space in human ventral temporal cortex. Haxby JV, Guntupalli JS, Connolly AC, Halchenko YO, Conroy BR, Gobbini MI, Hanke M, Ramadge PJ. Neuron. 2011 Oct 20;72(2):404-16. - High dimensional mapping of different subjects’ voxel spaces onto each other

If neural activation patterns don’t capture regularities, then what will? We need to abstract away from subject-specific neural fingerprints of activation. But abstract towards what? Previous neural decoding studies: To do neural decoding, take the subjects’ neural activation and enter it into a decoder A different approach: Operate not on neural activation patterns, but instead on the similarity relations between those activation patterns Neural fingerprints are subject-specific. In order to find population-level regularities, we need to abstract away from them

Hypotheses: What will we gain by operating in similarity space? Neural similarity will: Reveal the structure of people’s representations Reveal population-level regularities in that structure

How to decode in similarity-space: a simple solution Suppose A, B and C are cities, and the similarities represent geographical distances Which cities correspond to A, B and C? NYC San Diego Boston A B C

Across-subject decoding via neural similarity-space Goal: Figure out which labels correspond to which stimulus conditions, using only condition-labels from the other subjects Neural similarity: Spatial correlation between activation patterns Raizada & Connolly, J.Cog.Neuro, 2012

A simple solution: permute the labels, and see which permutation matches best Raizada & Connolly, J.Cog.Neuro, 2012

How well does this work with actual neural data? Dataset: Haxby et al, Science, 2001. “Distributed and overlapping representations of faces and objects in ventral temporal cortex.” http://dev.pymvpa.org/datadb/haxby2001.html Six subjects, eight stimulus categories: bottles, cats, chairs, faces, houses, scissors, scrambled-pictures, shoes Animate and inanimate objects Voxels from ventral-temporal (VT) cortex masks Lingual, parahippocampal, fusiform, and inferior temporal gyri Raizada & Connolly, J.Cog.Neuro, 2012

How well does this work with actual neural data? Result: 91.7% correct 44 out of 48 decodings correct 48 = 6x8: 6 subjects, 8 categories per subj Five subjects: all eight categories correct One subject: 4 out of 8 correct Confusions: bottle-scissors, shoe-chair There are 8 categories, so chance level is 1/8 Chance-level is 12.5% correct Raizada & Connolly, J.Cog.Neuro, 2012

Striking convergence with a proposal from the philosophy of mind P.M. Churchland (1998) “Conceptual similarity across sensory and neural diversity: The Fodor/Lepore challenge answered”, J. Philosophy, 95, 5-32.