Fast Readout of Object Identity from Macaque Inferior Tempora Cortex Chou P. Hung, Gabriel Kreiman, Tomaso Poggio, James J.DiCarlo McGovern Institute for Brain Research, Brain and Cognitive Sciences, MIT

Object Recognition is difficult: trade-off between selectivity and invariance Selectivity Many different images can correspond to the same type of object Invariance Similar activation patterns can correspond to different objects

The end station of the ventral stream in visual cortex is IT

Can we readout what the monkey is seeing?

Single electrode recordings Anterior inferior temporal cortex: highest visual area in the ventral “what” pathway Spiking activity in AIT shows selectivity for complex shapes

Can we “read-out” the subject’s object percept from IT? number of sites for reliable, real-time performance temporal properties (onset + integration scale) of object information neural code for different tasks invariance to object position, size, pose, illumination, clutter recognition: ‘classification’ vs. ‘identification’? spatial scale of object information (single unit, multi- unit, LFP) stability of these neuronal codes? improvement with experience? …

77 objects, 8 classes

Recording at each recording site during passive viewing 77 visual objects 10 presentation repetitions per object presentation order randomized and counter- balanced

One-versus-all classification g classes ( g =8): G 1, …, G g (toys, monkey faces, vehicles, etc.) For each class i, build a binary classifier f i (toys vs. rest, monkey faces vs. rest, etc.) s j labeled examples (j=1,…,n), For each example j, compute the output of each classifier ( e.g. p i =s j ． f i ) Take prediction that maximizes p i One-versus-all is not worse than other methods (Rifkin et al, 2003)

Comparison of different statistical classifiers

Decoding the population response Categorization 8 groups

Pattern of mistakes made by the classifier

Very rapid read-out of object information

Categorization and Identification

IT representation is invariant to changes in position and size

Neural code in IT: time resolution

Neural code in IT: latency and integration time

Reading out another type of object info: scale and location

How are different kinds of information coded?

Reading out another type of object info: stimulus onset

Specific wiring significantly improves classifier performance

Extrapolation to novel pictures within the same categories

Strong overlap between the best neurons for categorization and identification

The SNR for categorization and identification are positively correlated

Invariance to scale and position