1. Learning and Tuning of Neurons in Inferior Temporal Cortex
Learning and Neural Plasticity in the Adult Visual System Society for Neuroscience San Diego, California
Bharathi Jagadeesh Department of Physiology & Biophysics University of Washington Seattle, Washington

2. Macaque temporal lobe PG PF V1 TE Ventral or “What” processing stream
TEO TE

3. Pictures of people, places, and things

4. best
next best
100 s/s
300 ms
Copied from algorithm paper figs.
worst

5. What does the selectivity in IT mean?

6. Perceptual similarityV1 V2 V4
TEO IT
Perceptual similarity IT
response

7. Perceptual similarity
Image characteristics
Experience

8. Similarity of stimuli should explain selectivity in IT cortex
Perceptual similarity
Proposed relationship
Neural responses in IT

9. Measuring perceptual similarity
Neural responses in IT
Proposed relationship
Perceptual similarity algorithms

10. How do we use perceptual similarity algorithms?

11. wang3.eps file contains actual image numbers This is a 10X10 image grid created by the imgrandom command

12. Image similarity algorithms
SIMPlicity algorithm
Wang et al (2001)
Image divided into 4x4 pixel blocks, feature vector is calculated for each block.
Feature vector 6-dimensional: Color dimensions, (LUV space, 3 dimensions) , Spatial frequency, wavelet analysis on the L component of the image (3 dimensions)
Number of regions using a k-means algorithm.
The similarity between two images computed by comparing regions using Integrated Region Matching (IRM).
The SIMPLIcity (similarity) distance is weighted sum of similarity between regions.
Copied from algorithm paper figs.

13. Calculate image distances between images
25.8
32.2
38.4
33.5
59.9
47.5
31.2
57.9
30.0
40.9
54.9
32.1
Copied from algorithm paper figs.

14. Prediction ? Perceptual similarity Proposed relationship
Perceptual similarity algorithms
Neural responses in IT

15. Random images (24) using imgrandom command. In wang1
Random images (24) using imgrandom command. In wang1.eps, 2nd random seed (I.e. run imgrandom twice).

16. 3.54
4.07
4.34
4.92
5.13
5.24
5.56
5.72
5.90
5.93
6.05
6.39
6.42
6.49
7.00
7.12
7.33
7.34
7.49
7.50
7.51
52038 image (from previous slide) used as target to search. Created in matlab, using imgLowEMD (WANG), some text deleted. Wang distances shown.

17. Again, as search image, imglowWang used to plot data, but in 10X10 grid. Again, wang distances.

18. SIMPlicity retrieves targets
1.00
0.75
0.5
Precision
Copied from Sarah’s creation, in contrast.ppt file is the example image.
0.25 50
100
150
200
Number of Relevant Images Retrieved

19. Algorithms predict perceptual similarity
Proposed relationship
Perceptual similarity algorithms ?
Neural responses in IT

20. Do perceptual similarity algorithms explain neural responses in IT cortex?

21. best
next best
100 s/s
300 ms
Copied from algorithm paper figs
worst

22. Example cell: image distance between best/next and best/worst45
best
best 30
SIMPLIcity
next best
worst 15
Copied from algorithm paper figs
best-next best
best-worst

23. Population: Distance between Best-worst v. Best-Next Best50
100
best 50 40
worst 30
( best and worst stimuli)
SIMPLIcity
best-next
best-worst
Copied from algorithm paper figs, xl graph from xl file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls
best
next best
SIMPLIcity
(best and next best stimuli)

24. Do other similarity algorithms explain neural responses in IT cortex?

25. Contrast: Doesn’t retrieve targets50
100
150
200
0.25
0.50
0.75
1.00
Number of Relevant Images Retrieved
Precision
From sarah, contrast.ppt file

26. And, doesn’t explain IT responses
0.0
0.2
0.4
0.6
best
0.2
0.1
worst
RMS contrast difference
(best and worst stimuli)
best-next
best-worst
Copied from algorithm paper figs, xl graph from xl file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls
best
next best
RMS contrast difference
(best and next best stimuli)

27. EMD, another similarity metric: Retrieves targets50
100
150
200
0.25
0.50
0.75
1.00
Number of Relevant Images Retrieved
Precision
From sarah, contrast.ppt file

28. And, also explains IT responses50
100 20 40 60 80 40 30 20
(best and worst stimuli)
EMD
best-next
best-worst
Copied from algorithm paper figs, xl graph from xl file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls
best
next best
EMD
(best and next best stimuli)

29. Prediction Behavior Perceptual similarity ? Proposed relationship
Perceptual similarity algorithms
Figure from file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls
Neural responses in IT

30. Delayed match to sample (DMS) (easy pair)
Fixation ms
Stimulus
16-512ms
Mask
256 ms
Delay ms
Delay ms
Response
Copied from figures for thesis proposal, then adjusted the size, images replaced by easy image pair from gabe’ behavior data, and the cor_beh_neur4_bj.xls file

31. DMS (difficult pair)
Fixation ms
Stimulus
16-512ms
Mask
256 ms
Delay ms
Delay ms
Response
Copied from figures for thesis proposal, then adjusted the size, images replaced by easy image pair from gabe’ behavior data, and the cor_beh_neur4_bj.xls file

32. Measure “perceptual similarity”
Low performance
High performance
71%
96%
50 ms stimulus presentation
Again, images, percent correct from gabe’ behavior data, and the cor_beh_neur4_bj.xls file

33. Measure neural selectivity
Low performance
High performance
71%
96%
50 ms stimulus presentation
Copied from gabe’ behavior data, and the cor_beh_neur4_bj.xls file, linus’s neural data
62%
86%
Average neural response difference in passive fixation task

34. Neural performance v Behavior1
r = 0.57
0.9
Neural ROC
0.8
From gabe’ behavior data, and the cor_beh_neur4_bj.xls file and linus’s neural data. The calculation is complicated and not entierely replacable the way I have it stored.
0.7
0.6
0.6
0.7
0.8
0.9 1
Behavioral performance

35. Prediction Perceptual similarity Perceptual similarity algorithms
Neural responses in IT

36. Individual correlations
0.6
0.4
Correlation r
0.2
gabe-lin-beh-sfn2004.xls
Behavior v neuron
Algorithm v
neuron

37. Prediction Perceptual similarity Perceptual similarity algorithms
Neural responses in IT

38. Partial correlations Partial correlation r 0.6 0.4 0.2
gabe-lin-beh-sfn2004.xls (but, also uses cor_beh xl file, and the calcs actually done at a web site)
Behavior v neuron
Algorithm v
neuron

39. Perceptual similarity correlated with IT neuron response similarity
Perceptual similarity algorithms
Neural responses in IT

40. How does training change the relationship to the algorithm?

41. Passive Association Task
Fixation
Predictor
Delay
Choice
Bar
release Go
Images from swass analysis figures, format from cynthia’s association task
No Go
Erickson CA, Desimone R (1999)

42. Valid Association Trials
Go Trials
No Go Trials
Predictor
Choice
Predictor
Choice
Go 359 ms
No Go
Go 377 ms
No Go
Images from swass analysis figures, format from cynthia’s association task
swass146.itm file,
98.7% correct valid trials
100% correct invalid trials
Latency valid trials ms
Latency invalid trials 437.1ms
Go 376 ms
No Go
Go 369 ms
No Go

43. Invalid Association Trials
Go Trials
No Go Trials
Predictor
Target
Predictor
Target
Go 359/473 ms
No Go
Go 377/465 ms
No Go
Images from swass analysis figures, format from cynthia’s association task
swass146.itm file,
98.7% correct valid trials
100% correct invalid trials
Latency valid trials ms
Latency invalid trials 437.1ms
Go 376/436 ms
No Go
Go 369/363 ms
No Go

44. Response to choice stimulus is correlated with response to predictor
Neural response to choice
Images from swass analysis figures, format from cynthia’s association task
swass146.itm file,
98.7% correct valid trials
100% correct invalid trials
Latency valid trials ms
Latency invalid trials 437.1ms
Erickson & Desimone (1999)
Neural response to predictor

45. Dissimilar stimuli produce similar responses
Images from swass analysis figures, format from cynthia’s association task
swass146.itm file,
98.7% correct valid trials
100% correct invalid trials
Latency valid trials ms
Latency invalid trials 437.1ms
Erickson & Desimone (1999)

46. Training breaks relationship to algorithm35 37 39 41 43
Similar responses
Different responses
Image distance 35 37 39 41 43
Similar Responses
Different
responses
Image distance
gabe-lin-beh-sfn2004.xls, but data copied from that file as well as wang_numbers_sra.xls
Passive fixation: No training
Association task: Training
Data from Erickson & Desimone (1999)

47. Specific conclusions
Perceptual similarity algorithms measure (at least partially) the perceptual similarity of stimuli.
The same algorithms explain (at least partially) the neural response similarity in IT.
But, neural response similarity is better correlated with discrimination performance (measured perceptual similarity) than it is with the image similarity algorithms.
And, training that modifies the processing of stimuli breaks the relationship between image similarity and neural response similarity.

48. Jagadeesh lab University of Washington
Katie Ahl Sarah Allred Yan Liu, M.D., Ph.D. Jen Skiver Thompson Andrew Derrington, Ph.D. Cynthia Erickson, Ph.D. J.M.Jagadeesh, Ph.D. Amanda Parker, Ph.D.
Jamie Bullis Rebecca Mease Amber McAlister
Current members
Visiting scholars and collaborators
Rotation students & former members

49. Divider
What comes after isn’t supposed to be included right now.

50. Methods
Record from single neurons in the non-human primate brain, while the primate performs visual tasks.
Monitor eye movements so that visual stimulus at the retina is known.
In my lab, we record from inferotemporal cortex in the macaque, and are of cortex thought to be important for perception of people places and things.

51. best
next best
100 s/s
300 ms
Copied from algorithm paper figs
worst

52. Population within v across groups50
100 B 43 41
Image distance 39
(effective v ineffective groups)
SIMPLIcity 37 35
Eff
Eff-Ineff
Copied from algorithm paper figs, xl graph from xl file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls
SIMPLIcity
(within effective group)

53. Histograms, population, sorted by emd rank to best
100
200
300
400
500
600 10 20 30 40
time in ms after stimulus onset
spikes per second
rank 1-8
rank 10-17
rank 18-24

54. Population: all comparisons-1
-0.5
0.5 1 20 40 60 80
r-value
frequency

55. Example : SIMPlicity correlated with neural response
best
worst 80 80 60 60
neural response (spikes/second)
neural response (spikes/second) 40 40 20 20 20 40 60 80 20 40 60 80
SIMPLIcity to best
SIMPLIcity to worst

56. Population: SIMPlicity is correlated with neural response80 80
best
frequency 60 60 40 40 20 20 80 80
worst 60 60
frequency 40 40 20 20 -1
-0.5
0.5 1
-100
-50 50
100
R-value
Slope

57. Population: correlation to each
0.15
SIMPLIcity
shuffle
0.12
0.09
Average r-value (absolute value)
0.06
0.03 5 10 15 20 25
Stimulus Rank (best to worst)

58. EMD: Does retrieve images, also explains neural responseB 50
100 20 40 60 80
EMD
(best and next best stimuli)
(best and worst stimuli) 50
100
150
200
0.25
0.5
0.75 1
Number of Relevant Images Retrieved
Precision
avg EMD
example EMD
chance

59. Copied from algorithm paper figs.

61. Individual correlations
0.6
0.4
corelation, r
0.2
0.0
neuron/behavior
Image sim/neuron
Image sim/behavior

62. Individual correlations
0.6
0.4
0.2
beh/neu
emd/beh
emd/neu

63. Partial correlations
0.2
0.4
0.6
beh/neu
img/beh
img/neu

64. Invalid v valid latencies
700
600
invalid latencies (ms)
500
400
Data copied from swass-anal-bj xls
300
300
400
500
600
700
valid latencies (ms)

65. Valid v Invalid trial latencies
700
600
10% of trials
invalid latencies (ms)
500
400
300
Data copied from swass-anal-bj xls (swass0147, copied from swass-experiment.ppt)
300
400
500
600
700
valid latencies (ms)
90% of trials

66. With training, IT relationship to algorithmic image similarity breaks50
100 10 20 30 40 50 60
within effective
across
across
Copied from this file (from algorithm paper figures) -- left graph. Copied from r ppt files for cynthia’s data.
within effective
Data from Erickson and Desimone (1999)

67. V1 V2 V4
TEO IT V1
response
orientation

68. Proposal: IT neurons represent the perceptual similarity of stimuli.

69. Prediction Behavior Perceptual similarity ? Proposed relationship
Perceptual similarity algorithms
Figure from file gabe-lin-beh-sfn.xls, which contains numbers drawn from wang_numbers_sra.xls
Retrieval
0.1
0.2
0.5 1
Image diff
Neural responses in IT

70. Characteristics of algorithms
Rely heavily on color content of images
Pairwise comparison, not a parametric space
Ignores “cognitive” information
But, works for realistic images, because images that are similar to one another in the real world tend to share patterns of colors.

71. Algorithms predict perceptual similarity
Test of algorithm in “spiked” databases.
Correlation between algorithm and human sorting of images.
Correlation between algorithm and monkey performance in discrimination task.