Learning and Tuning of Neurons in Inferior Temporal Cortex Learning and Neural Plasticity in the Adult Visual System Society for Neuroscience San Diego, California Bharathi Jagadeesh Department of Physiology & Biophysics University of Washington Seattle, Washington

Macaque temporal lobe PG PF V1 TE Ventral or “What” processing stream TEO TE

Pictures of people, places, and things

best next best 100 s/s 300 ms Copied from algorithm paper figs. worst

What does the selectivity in IT mean?

Perceptual similarity V1 V2 V4 TEO IT Perceptual similarity IT response

Perceptual similarity Image characteristics Experience

Similarity of stimuli should explain selectivity in IT cortex Perceptual similarity Proposed relationship Neural responses in IT

Measuring perceptual similarity Neural responses in IT Proposed relationship Perceptual similarity algorithms

How do we use perceptual similarity algorithms?

wang3.eps file contains actual image numbers This is a 10X10 image grid created by the imgrandom command

Image similarity algorithms SIMPlicity algorithm Wang et al (2001) Image divided into 4x4 pixel blocks, feature vector is calculated for each block. Feature vector 6-dimensional: Color dimensions, (LUV space, 3 dimensions) , Spatial frequency, wavelet analysis on the L component of the image (3 dimensions) Number of regions using a k-means algorithm. The similarity between two images computed by comparing regions using Integrated Region Matching (IRM). The SIMPLIcity (similarity) distance is weighted sum of similarity between regions. Copied from algorithm paper figs.

Calculate image distances between images 25.8 32.2 38.4 33.5 59.9 47.5 31.2 57.9 30.0 40.9 54.9 32.1 Copied from algorithm paper figs.

Prediction ? Perceptual similarity Proposed relationship Perceptual similarity algorithms Neural responses in IT

Random images (24) using imgrandom command. In wang1 Random images (24) using imgrandom command. In wang1.eps, 2nd random seed (I.e. run imgrandom twice).

3.54 4.07 4.34 4.92 5.13 5.24 5.56 5.72 5.90 5.93 6.05 6.39 6.42 6.49 7.00 7.12 7.33 7.34 7.49 7.50 7.51 52038 image (from previous slide) used as target to search. Created in matlab, using imgLowEMD (WANG), some text deleted. Wang distances shown.

Again, 52038 as search image, imglowWang used to plot data, but in 10X10 grid. Again, wang distances.

SIMPlicity retrieves targets 1.00 0.75 0.5 Precision Copied from Sarah’s creation, in contrast.ppt file. 53028 is the example image. 0.25 50 100 150 200 Number of Relevant Images Retrieved

Algorithms predict perceptual similarity Proposed relationship Perceptual similarity algorithms ? Neural responses in IT

Do perceptual similarity algorithms explain neural responses in IT cortex?

best next best 100 s/s 300 ms Copied from algorithm paper figs worst

Example cell: image distance between best/next and best/worst 45 best best 30 SIMPLIcity next best worst 15 Copied from algorithm paper figs best-next best best-worst

Population: Distance between Best-worst v. Best-Next Best 50 100 best 50 40 worst 30 ( best and worst stimuli) SIMPLIcity best-next best-worst Copied from algorithm paper figs, xl graph from xl file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls best next best SIMPLIcity (best and next best stimuli)

Do other similarity algorithms explain neural responses in IT cortex?

Contrast: Doesn’t retrieve targets 50 100 150 200 0.25 0.50 0.75 1.00 Number of Relevant Images Retrieved Precision From sarah, contrast.ppt file

And, doesn’t explain IT responses 0.0 0.2 0.4 0.6 best 0.2 0.1 worst RMS contrast difference (best and worst stimuli) best-next best-worst Copied from algorithm paper figs, xl graph from xl file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls best next best RMS contrast difference (best and next best stimuli)

EMD, another similarity metric: Retrieves targets 50 100 150 200 0.25 0.50 0.75 1.00 Number of Relevant Images Retrieved Precision From sarah, contrast.ppt file

And, also explains IT responses 50 100 20 40 60 80 40 30 20 (best and worst stimuli) EMD best-next best-worst Copied from algorithm paper figs, xl graph from xl file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls best next best EMD (best and next best stimuli)

Prediction Behavior Perceptual similarity ? Proposed relationship Perceptual similarity algorithms Figure from file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls Neural responses in IT

Delayed match to sample (DMS) (easy pair) Fixation 250-500ms Stimulus 16-512ms Mask 256 ms Delay 100-500ms Delay 500-1200ms Response Copied from figures for thesis proposal, then adjusted the size, images replaced by easy image pair from gabe’ behavior data, and the cor_beh_neur4_bj.xls file

DMS (difficult pair) Fixation 250-500ms Stimulus 16-512ms Mask 256 ms Delay 100-500ms Delay 500-1200ms Response Copied from figures for thesis proposal, then adjusted the size, images replaced by easy image pair from gabe’ behavior data, and the cor_beh_neur4_bj.xls file

Measure “perceptual similarity” Low performance High performance 71% 96% Performance @ 50 ms stimulus presentation Again, images, percent correct from gabe’ behavior data, and the cor_beh_neur4_bj.xls file

Measure neural selectivity Low performance High performance 71% 96% Performance @ 50 ms stimulus presentation Copied from gabe’ behavior data, and the cor_beh_neur4_bj.xls file, linus’s neural data 62% 86% Average neural response difference in passive fixation task

Neural performance v Behavior 1 r = 0.57 0.9 Neural ROC 0.8 From gabe’ behavior data, and the cor_beh_neur4_bj.xls file and linus’s neural data. The calculation is complicated and not entierely replacable the way I have it stored. 0.7 0.6 0.6 0.7 0.8 0.9 1 Behavioral performance

Prediction Perceptual similarity Perceptual similarity algorithms Neural responses in IT

Individual correlations 0.6 0.4 Correlation r 0.2 gabe-lin-beh-sfn2004.xls Behavior v neuron Algorithm v neuron

Prediction Perceptual similarity Perceptual similarity algorithms Neural responses in IT

Partial correlations Partial correlation r 0.6 0.4 0.2 gabe-lin-beh-sfn2004.xls (but, also uses cor_beh xl file, and the calcs actually done at a web site) Behavior v neuron Algorithm v neuron

Perceptual similarity correlated with IT neuron response similarity Perceptual similarity algorithms Neural responses in IT

How does training change the relationship to the algorithm?

Passive Association Task Fixation Predictor Delay Choice Bar release Go Images from swass analysis figures, format from cynthia’s association task No Go Erickson CA, Desimone R (1999)

Valid Association Trials Go Trials No Go Trials Predictor Choice Predictor Choice Go 359 ms No Go Go 377 ms No Go Images from swass analysis figures, format from cynthia’s association task swass146.itm file, 98.7% correct valid trials 100% correct invalid trials Latency valid trials 370.6 ms Latency invalid trials 437.1ms Go 376 ms No Go Go 369 ms No Go

Invalid Association Trials Go Trials No Go Trials Predictor Target Predictor Target Go 359/473 ms No Go Go 377/465 ms No Go Images from swass analysis figures, format from cynthia’s association task swass146.itm file, 98.7% correct valid trials 100% correct invalid trials Latency valid trials 370.6 ms Latency invalid trials 437.1ms Go 376/436 ms No Go Go 369/363 ms No Go

Response to choice stimulus is correlated with response to predictor Neural response to choice Images from swass analysis figures, format from cynthia’s association task swass146.itm file, 98.7% correct valid trials 100% correct invalid trials Latency valid trials 370.6 ms Latency invalid trials 437.1ms Erickson & Desimone (1999) Neural response to predictor

Dissimilar stimuli produce similar responses Images from swass analysis figures, format from cynthia’s association task swass146.itm file, 98.7% correct valid trials 100% correct invalid trials Latency valid trials 370.6 ms Latency invalid trials 437.1ms Erickson & Desimone (1999)

Training breaks relationship to algorithm 35 37 39 41 43 Similar responses Different responses Image distance 35 37 39 41 43 Similar Responses Different responses Image distance gabe-lin-beh-sfn2004.xls, but data copied from that file as well as wang_numbers_sra.xls Passive fixation: No training Association task: Training Data from Erickson & Desimone (1999)

Specific conclusions Perceptual similarity algorithms measure (at least partially) the perceptual similarity of stimuli. The same algorithms explain (at least partially) the neural response similarity in IT. But, neural response similarity is better correlated with discrimination performance (measured perceptual similarity) than it is with the image similarity algorithms. And, training that modifies the processing of stimuli breaks the relationship between image similarity and neural response similarity.

Jagadeesh lab University of Washington Katie Ahl Sarah Allred Yan Liu, M.D., Ph.D. Jen Skiver Thompson Andrew Derrington, Ph.D. Cynthia Erickson, Ph.D. J.M.Jagadeesh, Ph.D. Amanda Parker, Ph.D. Jamie Bullis Rebecca Mease Amber McAlister Current members Visiting scholars and collaborators Rotation students & former members

Divider What comes after isn’t supposed to be included right now.

Methods Record from single neurons in the non-human primate brain, while the primate performs visual tasks. Monitor eye movements so that visual stimulus at the retina is known. In my lab, we record from inferotemporal cortex in the macaque, and are of cortex thought to be important for perception of people places and things.

best next best 100 s/s 300 ms Copied from algorithm paper figs worst

Population within v across groups 50 100 B 43 41 Image distance 39 (effective v ineffective groups) SIMPLIcity 37 35 Eff Eff-Ineff Copied from algorithm paper figs, xl graph from xl file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls SIMPLIcity (within effective group)

Histograms, population, sorted by emd rank to best 100 200 300 400 500 600 10 20 30 40 time in ms after stimulus onset spikes per second rank 1-8 rank 10-17 rank 18-24

Population: all comparisons -1 -0.5 0.5 1 20 40 60 80 r-value frequency

Example : SIMPlicity correlated with neural response best worst 80 80 60 60 neural response (spikes/second) neural response (spikes/second) 40 40 20 20 20 40 60 80 20 40 60 80 SIMPLIcity to best SIMPLIcity to worst

Population: SIMPlicity is correlated with neural response 80 80 best frequency 60 60 40 40 20 20 80 80 worst 60 60 frequency 40 40 20 20 -1 -0.5 0.5 1 -100 -50 50 100 R-value Slope

Population: correlation to each 0.15 SIMPLIcity shuffle 0.12 0.09 Average r-value (absolute value) 0.06 0.03 5 10 15 20 25 Stimulus Rank (best to worst)

EMD: Does retrieve images, also explains neural response B 50 100 20 40 60 80 EMD (best and next best stimuli) (best and worst stimuli) 50 100 150 200 0.25 0.5 0.75 1 Number of Relevant Images Retrieved Precision avg EMD example EMD chance

Copied from algorithm paper figs.

25.8 17.7 32.2 19.3 38.4 29.2 33.5 24.9 59.9 37.8 47.5 30.4 31.2 25.1 57.9 35.7 30.0 22.3 40.9 31.5 54.9 39.0 32.1 25.6

Individual correlations 0.6 0.4 corelation, r 0.2 0.0 neuron/behavior Image sim/neuron Image sim/behavior

Individual correlations 0.6 0.4 0.2 beh/neu emd/beh emd/neu

Partial correlations 0.2 0.4 0.6 beh/neu img/beh img/neu

Invalid v valid latencies 700 600 invalid latencies (ms) 500 400 Data copied from swass-anal-bj-040928.xls 300 300 400 500 600 700 valid latencies (ms)

Valid v Invalid trial latencies 700 600 10% of trials invalid latencies (ms) 500 400 300 Data copied from swass-anal-bj-040928.xls (swass0147, copied from swass-experiment.ppt) 300 400 500 600 700 valid latencies (ms) 90% of trials

With training, IT relationship to algorithmic image similarity breaks 50 100 10 20 30 40 50 60 within effective across across Copied from this file (from algorithm paper figures) -- left graph. Copied from r01-2003.ppt files for cynthia’s data. within effective Data from Erickson and Desimone (1999)

V1 V2 V4 TEO IT V1 response orientation

Proposal: IT neurons represent the perceptual similarity of stimuli.

Prediction Behavior Perceptual similarity ? Proposed relationship Perceptual similarity algorithms Figure from file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls Retrieval 0.1 0.2 0.5 1 Image diff Neural responses in IT

Characteristics of algorithms Rely heavily on color content of images Pairwise comparison, not a parametric space Ignores “cognitive” information But, works for realistic images, because images that are similar to one another in the real world tend to share patterns of colors.

Algorithms predict perceptual similarity Test of algorithm in “spiked” databases. Correlation between algorithm and human sorting of images. Correlation between algorithm and monkey performance in discrimination task.