1. Fast Readout of Object Identity from Macaque Inferior Tempora Cortex Chou P. Hung, Gabriel Kreiman, Tomaso Poggio, James J.DiCarlo McGovern Institute for Brain Research, Brain and Cognitive Sciences, MIT

2. Object Recognition is difficult: trade-off between selectivity and invariance Selectivity Many different images can correspond to the same type of object Invariance Similar activation patterns can correspond to different objects

3. The end station of the ventral stream in visual cortex is IT

4. Can we readout what the monkey is seeing?

5. Single electrode recordings Anterior inferior temporal cortex: highest visual area in the ventral “what” pathway Spiking activity in AIT shows selectivity for complex shapes

8. Can we “read-out” the subject’s object percept from IT? number of sites for reliable, real-time performance temporal properties (onset + integration scale) of object information neural code for different tasks invariance to object position, size, pose, illumination, clutter recognition: ‘classification’ vs. ‘identification’? spatial scale of object information (single unit, multi- unit, LFP) stability of these neuronal codes? improvement with experience? …

9. 77 objects, 8 classes

10. Recording at each recording site during passive viewing 77 visual objects 10 presentation repetitions per object presentation order randomized and counter- balanced

11. One-versus-all classification g classes ( g =8): G 1, …, G g (toys, monkey faces, vehicles, etc.) For each class i, build a binary classifier f i (toys vs. rest, monkey faces vs. rest, etc.) s j labeled examples (j=1,…,n), For each example j, compute the output of each classifier ( e.g. p i =s j ． f i ) Take prediction that maximizes p i One-versus-all is not worse than other methods (Rifkin et al, 2003)

12. Comparison of different statistical classifiers

13. Decoding the population response Categorization 8 groups

14. Pattern of mistakes made by the classifier

15. Very rapid read-out of object information

16. Categorization and Identification

17. IT representation is invariant to changes in position and size

20. Neural code in IT: time resolution

21. Neural code in IT: latency and integration time

22. Reading out another type of object info: scale and location

23. How are different kinds of information coded?

24. Reading out another type of object info: stimulus onset

25. Specific wiring significantly improves classifier performance

26. Extrapolation to novel pictures within the same categories

27. Strong overlap between the best neurons for categorization and identification

28. The SNR for categorization and identification are positively correlated

29. Invariance to scale and position