Physiology of Vision: a swift overview 16-721: Learning-Based Methods in Vision A. Efros, CMU, Spring 2007 Some figures from Steve Palmer
Class Introductions Name: Research area / project / advisor What you want to learn in this class? When I am not working, I ______________ Favorite fruit:
Image Formation Digital Camera Film The Eye
Monocular Visual Field: 160 deg (w) X 135 deg (h) Binocular Visual Field: 200 deg (w) X 135 deg (h)
What do we see? 3D world 2D image Figures © Stephen E. Palmer, 2002
What do we see? 3D world 2D image Painted backdrop
The Plenoptic Function Figure by Leonard McMillan Q: What is the set of all things that we can ever see? A: The Plenoptic Function (Adelson & Bergen) Let’s start with a stationary person and try to parameterize everything that he can see…
Grayscale snapshot P(q,f) is intensity of light Seen from a single view point At a single time Averaged over the wavelengths of the visible spectrum (can also do P(x,y), but spherical coordinate are nicer)
Color snapshot P(q,f,l) is intensity of light Seen from a single view point At a single time As a function of wavelength
Spherical Panorama All light rays through a point form a ponorama See also: 2003 New Years Eve http://www.panoramas.dk/fullscreen3/f1.html All light rays through a point form a ponorama Totally captured in a 2D array -- P(q,f) Where is the geometry???
A movie P(q,f,l,t) is intensity of light Seen from a single view point Over time As a function of wavelength
Space-time images t y x
Holographic movie P(q,f,l,t,VX,VY,VZ) is intensity of light Seen from ANY viewpoint Over time As a function of wavelength
The Plenoptic Function P(q,f,l,t,VX,VY,VZ) Can reconstruct every possible view, at every moment, from every position, at every wavelength Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality! Not bad for a function…
The Eye is a camera The human eye is a camera! Iris - colored annulus with radial muscles Pupil - the hole (aperture) whose size is controlled by the iris What’s the “film”? photoreceptor cells (rods and cones) in the retina
The Retina
Retina up-close Light
Two types of light-sensitive receptors Cones cone-shaped less sensitive operate in high light color vision Rods rod-shaped highly sensitive operate at night gray-scale vision © Stephen E. Palmer, 2002
Rod / Cone sensitivity The famous sock-matching problem…
Distribution of Rods and Cones Night Sky: why are there more stars off-center? © Stephen E. Palmer, 2002
Electromagnetic Spectrum At least 3 spectral bands required (e.g. R,G,B) Human Luminance Sensitivity Function http://www.yorku.ca/eye/photopik.htm
…because that’s where the Visible Light Why do we see light of these wavelengths? …because that’s where the Sun radiates EM energy © Stephen E. Palmer, 2002
The Physics of Light Any patch of light can be completely described physically by its spectrum: the number of photons (per time unit) at each wavelength 400 - 700 nm. © Stephen E. Palmer, 2002
The Physics of Light Some examples of the spectra of light sources © Stephen E. Palmer, 2002
The Physics of Light Some examples of the reflectance spectra of surfaces Red 400 700 Yellow 400 700 Blue 400 700 Purple 400 700 % Photons Reflected Wavelength (nm) © Stephen E. Palmer, 2002
The Psychophysical Correspondence There is no simple functional description for the perceived color of all lights under all viewing conditions, but …... A helpful constraint: Consider only physical spectra with normal distributions mean area variance © Stephen E. Palmer, 2002
The Psychophysical Correspondence Mean Hue # Photons Wavelength © Stephen E. Palmer, 2002
The Psychophysical Correspondence Variance Saturation Wavelength # Photons © Stephen E. Palmer, 2002
The Psychophysical Correspondence Area Brightness # Photons Wavelength © Stephen E. Palmer, 2002
Physiology of Color Vision Three kinds of cones: Why are M and L cones so close? © Stephen E. Palmer, 2002
Retinal Processing © Stephen E. Palmer, 2002
Single Cell Recording Microelectrode Electrical response Amplifier Electrical response (action potentials) mV © Stephen E. Palmer, 2002
Single Cell Recording © Stephen E. Palmer, 2002
Retinal Receptive Fields Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response © Stephen E. Palmer, 2002
Retinal Receptive Fields Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response © Stephen E. Palmer, 2002
Retinal Receptive Fields Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response © Stephen E. Palmer, 2002
Retinal Receptive Fields Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response © Stephen E. Palmer, 2002
Retinal Receptive Fields Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response © Stephen E. Palmer, 2002
Retinal Receptive Fields Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response © Stephen E. Palmer, 2002
Retinal Receptive Fields RF of On-center Off-surround cells © Stephen E. Palmer, 2002
Retinal Receptive Fields RF of Off-center On-surround cells Surround Center © Stephen E. Palmer, 2002
Retinal Receptive Fields
Retinal Receptive Fields Receptive field structure in bipolar cells Light © Stephen E. Palmer, 2002
Retinal Receptive Fields Receptive field structure in bipolar cells © Stephen E. Palmer, 2002
Visual Cortex Cortical Area V1 aka: Primary visual cortex Striate cortex Brodman’s area 17 © Stephen E. Palmer, 2002
Cortical Receptive Fields Single-cell recording from visual cortex David Hubel & Thorston Wiesel © Stephen E. Palmer, 2002
Cortical Receptive Fields Single-cell recording from visual cortex © Stephen E. Palmer, 2002
Cortical Receptive Fields Three classes of cells in V1 Simple cells Complex cells Hypercomplex cells © Stephen E. Palmer, 2002
Cortical Receptive Fields Simple Cells: “Line Detectors” © Stephen E. Palmer, 2002
Cortical Receptive Fields Simple Cells: “Edge Detectors” © Stephen E. Palmer, 2002
Cortical Receptive Fields Constructing a line detector © Stephen E. Palmer, 2002
Cortical Receptive Fields Complex Cells 0o © Stephen E. Palmer, 2002
Cortical Receptive Fields Complex Cells 60o © Stephen E. Palmer, 2002
Cortical Receptive Fields Complex Cells 90o © Stephen E. Palmer, 2002
Cortical Receptive Fields Complex Cells 120o © Stephen E. Palmer, 2002
Cortical Receptive Fields Constructing a Complex Cell © Stephen E. Palmer, 2002
Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002
Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002
Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002
Cortical Receptive Fields Hypercomplex Cells “End-stopped” Cells © Stephen E. Palmer, 2002
Cortical Receptive Fields “End-stopped” Simple Cells © Stephen E. Palmer, 2002
Cortical Receptive Fields Constructing a Hypercomplex Cell © Stephen E. Palmer, 2002
Mapping from Retina to V1
Why edges? So, why “edge-like” structures in the Plenoptic Function?
Because our world is structured!
Problem: Dynamic Range The real world is High dynamic range 1 1500 25,000 400,000 2,000,000,000
Is Camera a photometer? Image pixel (312, 284) = 42 42 photos? 2
Long Exposure 10-6 High dynamic range 106 10-6 106 0 to 255 Real world Picture 0 to 255
Short Exposure 10-6 High dynamic range 106 10-6 106 0 to 255 Real world 10-6 106 Picture 0 to 255
Varying Exposure
What does the eye sees? The eye has a huge dynamic range Do we see a true radiance map?
Eye is not a photometer! "Every light is a shade, compared to the higher lights, till you come to the sun; and every shade is a light, compared to the deeper shades, till you come to the night." — John Ruskin, 1879
Cornsweet Illusion
Campbell-Robson contrast sensitivity curve Sine wave Campbell-Robson contrast sensitivity curve
Metamers Eye is sensitive to changes (more on this later…)