Cortical circuits for vision Jamie Mazer Neurobiology of Cortical Systems Lecture 7 March 12, 2012

Readings for Thursday

How much of cortex is visual? (in primates) Van Essen flat map of macaque cortex Primates are likely an extreme example or an upper bound..

How much of cortex is visual? Van Essen flat map of macaque cortex simplified Felleman & Van Essen hierarchy

Key concepts phenomenon vs. implementation vs. function centrally synthesized maps –everything we perceive must be encoded by the retina –if so, whats all that visual cortex doing? –generating explicit sensory representations –emergent properties seem to be a key feature of high-level sensory cortical function –Question: Is cortex required to generate explicit or abstract properties? –Answer: Whats emergent in the retina? What about animals with not cortex, like birds and fish? are there common motifs across sensory modalities? – computational maps in other modalites? – what about other species? are they unique to cortex?

Retinal bipolar cells receptive fields

Retinal ganglion cell RFs (only retinal output)

Receptive fields and center-surround opponency

Center-surround organization Observed phenomenon? Implementation? Function?

Receptive fields and center-surround opponency Center-surround organization Observed phenomenon? Characteristic RF structure Implementation? Lateral inhibition Function? Spatial derivative; contrast enhancement

Behavioral consequences of center surround organization herring grid mach bands

Behavioral consequences of center surround organization herring grid mach bands

Thalamus: dLGN

What changes between the photoreceptors and LGN? transition from receptor potentials to spiking center-surround spatial receptive fields color opponency (B-Y/R-G) instead of simple cone- based wavelength tuning segregation into parallel processing streams –sustained and transient –fast and slow –on and off channels –color and luminance

Which brings us to primary visual cortex (BA 17; V1) m visual association primary visual

Topographic organization of V1 - retinotopy - orientation columns - occular dominance columns - non-oriented blobs (L2) - orientation topography

Thalamocortical projections and the canonical microcircuit

Primary visual cortex: simple cell orientation tuning hubel & wiesel 1968 orientation tuned V1 neuron MOVIE

Primary visual cortex: simple cell orientation tuning hubel & wiesel 1968 orientation tuned V1 neuron hubel & wiesel model

Primary visual cortex: simple cell orientation tuning hubel & wiesel 1968 orientation tuned V1 neuron hubel & wiesel model Key failures for the feedforward model? - contrast invariant orientation tuning

Primary visual cortex: simple cell orientation tuning hubel & wiesel 1968 orientation tuned V1 neuron hubel & wiesel model Hubel & Wiesel Interpretation Observed phenomenon? preferred orientation Implementation? linear summation of LGN cells Function? feature detectors for edges

Primary visual cortex: simple cell orientation tuning hubel & wiesel 1968 orientation tuned V1 neuronhubel & wiesel model Spatial Vision Interpretation Observed phenomenon? preferred orientation Implementation? quasi-linear combination of LGN cells Function? spatiotemporal filtering

cells prefer light increments or decrements cells have orientation tuning cells have a width tuning cells have length tuning cells have speed tuning cells are feature detectors where the feature is a bar of a particular orientation, size and speed intuitively obvious, simple to understand, seems to imply obvious behavioral function cells prefer light increments or decrements cells have orientation tuning cells have spatial frequency tuning cells have temporal frequency tuning cells are half-wave rectified spatiotemporal filters (Gabors) requires some math chops to understand, but has predictive power Feature detector model spatial vision model

Primary visual cortex: spatial frequency tuning Robson, DeValois, Maffei etc..

Feature detector model spatial vision model cells prefer light increments or decrements cells have orientation tuning cells have a width tuning cells have length tuning cells have speed tuning cells are feature detectors where the feature is a bar of a particular orientation, size and speed intuitively obvious, simple to understand, seems to imply obvious behavioral function cells prefer light increments or decrements cells have orientation tuning cells have spatial frequency tuning cells have temporal frequency tuning cells are half-wave rectified spatiotemporal filters requires some math chops to understand, but has predictive power

Primary visual cortex: simple complex hubel & wiesel 1968 simple complex

Primary visual cortex: simple complex hubel & wiesel 1968 simple complex MOVIE

Primary visual cortex: simple complex hubel & wiesel 1968 simple complex hypercomplex +length tuning

Primary visual cortex: simple complex hubel & wiesel 1968 simple cells pool center-surround neurons to form orientation selectivity complex cells pool simple cells to become position or phase invariant. and turtles all the way down…

Complex cells and the F 1 /F 0 ratio cats monkeys Skottun et al, 1991

Whats the spatial vision model got to say?

Complex cells and the F 1 /F 0 ratio Skottun et al, 1991 cats monkeys Mechler & Ringach, 2002 is this all an artifact?

Reverse correlation and the spike triggered average Jones & Palmer, 1987

Reverse correlation and the spike triggered average Jones & Palmer, 1987

Reverse correlation and the spike triggered average Jones & Palmer, 1987

V1 neurons are Gabors and Gabors are optimal… Daugman, 1985

V1 neurons are Gabors and Gabors are optimal… Daugman, 1985

Where do Gabors come from and the efficient coding hypothesis Barlow, 1972

Where do Gabors come from and the efficient coding hypothesis

Vinje & Gallant, 2000

Where do Gabors come from and the efficient coding hypothesis Haider et al, 2010

What have we established? simple cells – simple cells are partially assembled from LGN afferents – one basic flavor: Gabor they are bar-detectors as well (glass half empty), but the Gabor-model seems like a more compact framework complex cells – complex cells are assembled from simple cells – strict dichotomy not likely, more likely is, thalamocortical direct recipient simple cells, and, cells that are a combination of simple and non-simple innputs coding in V1 – sparseness is a hallmark of an efficient code – simple cells can be learned by maximizing sparseness – sparseness in V1 is based on center-surround (intracortical) inhibitory interations – the neural representation is awful close to what the computer vision people call a wavelet or multiscale pyramid and is the basis for things like MPG and JPG compression… perhaps we need more data from more complex stimuli?

Reverse correlation, complex cells and natural scenes

Problems: 1.STA doesnt really work for natural (non-white) stimuli 2.the STA is just plain wrong for complex cells

Linear receptive field maps in early vision DeAngelis et al, 1995 still orientation tuned! wheres it coming from?

Reverse correlation, complex cells and natural scenes Problems: 1.STA doesnt really work for natural (non-white) stimuli 2.the STA is just plain wrong for complex cells

Reverse correlation, complex cells and natural scenes Problems: 1.STA doesnt really work for natural (non-white) stimuli 2.the STA is just plain wrong for complex cells Spike Triggered Covariance (STC)

What have we established? simple cells – simple cells are partially assembled from LGN afferents – one basic flavor: Gabor they are bar-detectors as well (glass half empty), but the Gabor-model seems like a more compact framework complex cells – complex cells are assembled from simple cells – strict dichotomy not likely, more likely is, thalamocortical direct recipient simple cells, and, cells that are a combination of simple and non-simple innputs coding in V1 – sparseness is a hallmark of an efficient code – simple cells can be learned by maximizing sparseness – sparseness in V1 is based on center-surround (intracortical) inhibitory interations – the neural representation is awful close to what the computer vision people call a wavelet or multiscale pyramid and is the basis for things like MPG and JPG compression… perhaps we need more data from more complex stimuli? – STRFs, STC, regression analysis, MII etc all provide new tools could complex cells and complex stimuli… what did I not talk about??

Direction selectivity hubel&wiesel 1968 MOVIE

Direction selectivity is Gabor-ish too (vs. Reichardt Detector) DeAngelis et al,

Disparity/depth tuning focal plane near far foveae G. Poggio et al MOVIE

Disparity too fits in the spatial vision view… Note: Complex cells see anti-correlated bars differently than correlated, not true for perception… Ohzawa et al, 1997

Is spatial vision everything? the high-dimensional Gabor filter model explains a lot of the neurophysiological and psychophysical data, but.. – finding the right dimensions is non-trivial as well see next week. – even when the dimensions are likely identified, its essentially a linear or quasi-linear model and doesnt explain a range of observed phenomena, even in V1…

Center-surround interactions in V1 – generally NOT accounted for by the standard spatial vision model. end-stopping, length-tuning, hypercomplexity (H&W) cross-orientation inhibition (Silito et al) divisive gain control (Carandini paper!) curvature processing (Dobbins & Zucker) target pop-out (Knierim & Van Essen) attention and/or figure segmentation (Lamme et al)

What are the emergent properties of V1? new features extracted –orientation –binocular disparity (depth) –direction selectivity –spatial frequency –color (really transformed) new maps –orientation (columns) –ocular dominance –segregation of color info in blobs

What have we established? simple cells – simple cells are partially assembled from LGN afferents – one basic flavor: Gabor they are bar-detectors as well (glass half empty), but the Gabor-model seems like a more compact framework complex cells – complex cells are assembled from simple cells – strict dichotomy not likely, more likely is, thalamocortical direct recipient simple cells cells that are a combination of simple and non-simple innputs coding in V1 – sparseness is a hallmark of an efficient code – simple cells can be learned by maximizing sparseness – sparseness in V1 is based on center-surround (intracortical) inhibitory interations – the neural representation is awful close to what the computer vision people call a wavelet or multiscale pyramid and is the basis for things like MPG and JPG compression… perhaps we need more data from more complex stimuli? – STRFs, STC, regression analysis, MII etc all provide new tools could complex cells and complex stimuli… V1 exhibits multiple emergent properties What happens when you lose V1?? How much of this interpretation is primate-centric?

Primate-centric view? Andermann et al., 2011 Neill & Stryker, 2010 Shuler & Bear, Rodents have striate and extrastriate analogues or homologues - Tuning is similar, but not identical - Extra-retinal effects seem more pronounced

Readings for Thursday