1. Physiology of Vision: a swift overview
Learning-Based Methods in Vision
A. Efros, CMU, Spring 2012
Some figures from Steve Palmer

2. Tale of Martians with an old PC

3. Understanding the Brain
Anatomy versus Physiology
Anatomy: The biological study of the
physical structure of organisms.
Physiology: The biological study of the
functional structure of organisms.
© Stephen E. Palmer, 2002
© Stephen E. Palmer, 2002

4. The Brain is tricky business
Aristotle thought it’s for cooling the blood
Localized or distributed?

5. Phrenology
One of the first application of statistics This is how bad Machine Learning got started…

6. Localized or Distributed?
Evidence from patients with partial brain damage
Lots of useful data from soldiers in the Russo-Japanese war
But also evidence for distributed nature of processing
E.g. [Lashley] showed “graceful degradation” of memory performance in rats

7. The Visual System Both eye and brain are required for
functional vision
Two kinds of blindness:
Normal blindness (eye dysfunction)
Cortical blindness (brain dysfunction)
© Stephen E. Palmer, 2002
© Stephen E. Palmer, 2002

8. The Big Picture The Visual System Eyes register optical information
Pathways to occipital cortex
Two pathways from V1
“What” pathway to temporal cortex
“Where” pathway to parietal cortex
Convergence on frontal cortex
© Stephen E. Palmer, 2002
© Stephen E. Palmer, 2002

9. Pathways to the Brain Anatomy of Pathway to Visual Cortex
© Stephen E. Palmer, 2002
© Stephen E. Palmer, 2002

10. Pathways to the Brain The Lateral Geniculate Nucleus (LGN)
Waystation in Thalamus
Projections from both eyes
Six layers
Projects to cortical area V1
© Stephen E. Palmer, 2002
© Stephen E. Palmer, 2002

11. Visual Cortex Map of Visual Areas in Cortex “Unfolded” view of
visual areas in the
macaque cortex
(sizes not to scale).
© Stephen E. Palmer, 2002
© Stephen E. Palmer, 2002

12. Visual Cortex What/Where Pathways Evidence from lesions
of monkey cortex
© Stephen E. Palmer, 2002

13. Evidence from Neuropsychology
Visual Cortex
What/Where Pathways
Evidence from Neuropsychology
Visual agnosia:
Inability to identify objects and/or people
Caused by damage to inferior (lower) temporal lobe
Disruption of the “what” pathway
Visual neglect:
Inability to see objects in the left visual field
Caused by damage to right parietal lobe
Disruption of the “where” pathway
© Stephen E. Palmer, 2002
© Stephen E. Palmer, 2002

14. Feature-based Pathways Hypothesis
Visual Cortex
Feature-based Pathways Hypothesis
Visual Features
Color
Shape
Depth
Motion
Featural Pathways
Separate neural pathways
in which different features are
processed.
© Stephen E. Palmer, 2002
© Stephen E. Palmer, 2002

15. The Gross Summary The visual system is composed of
many interactive functional parts:
Eye (optics of image formation)
Retina (light transduction)
LGN (waystation?)
Area V1 (hypercolumns)
Higher cortical areas (features)
Cortical pathways (what/where)
© Stephen E. Palmer, 2002
© Stephen E. Palmer, 2002

16. Image Formation
Digital Camera
Film
The Eye

17. Monocular Visual Field: 160 deg (w) X 135 deg (h) Binocular Visual Field: 200 deg (w) X 135 deg (h)

18. What do we see?
3D world
2D image
Figures © Stephen E. Palmer, 2002

19. What do we see?
3D world
2D image
Painted
backdrop

20. The Plenoptic Function
Figure by Leonard McMillan
Q: What is the set of all things that we can ever see?
A: The Plenoptic Function (Adelson & Bergen)
Let’s start with a stationary person and try to parameterize everything that he can see…

21. Grayscale snapshot P(q,f) is intensity of light
Seen from a single view point
At a single time
Averaged over the wavelengths of the visible spectrum
(can also do P(x,y), but spherical coordinate are nicer)

22. Color snapshot P(q,f,l) is intensity of light
Seen from a single view point
At a single time
As a function of wavelength

23. Spherical Panorama All light rays through a point form a ponorama
See also: 2003 New Years Eve
All light rays through a point form a ponorama
Totally captured in a 2D array -- P(q,f)
Where is the geometry???

24. A movie P(q,f,l,t) is intensity of light Seen from a single view point
Over time
As a function of wavelength

25. Space-time images t y x

26. Holographic movie P(q,f,l,t,VX,VY,VZ) is intensity of light
Seen from ANY viewpoint
Over time
As a function of wavelength

27. The Plenoptic Function
P(q,f,l,t,VX,VY,VZ)
Can reconstruct every possible view, at every moment, from every position, at every wavelength
Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality! Not bad for a function…

28. The Eye is a camera The human eye is a camera!
Iris - colored annulus with radial muscles
Pupil - the hole (aperture) whose size is controlled by the iris
What’s the “film”?
photoreceptor cells (rods and cones) in the retina

29. The Retina

30. Retina up-close
Light

31. Two types of light-sensitive receptors
Cones
cone-shaped
less sensitive
operate in high light
color vision
Rods
rod-shaped
highly sensitive
operate at night
gray-scale vision
© Stephen E. Palmer, 2002

32. Rod / Cone sensitivity
The famous sock-matching problem…

33. Distribution of Rods and Cones
Night Sky: why are there more stars off-center?
© Stephen E. Palmer, 2002

34. Electromagnetic Spectrum
At least 3 spectral bands required (e.g. R,G,B)
Human Luminance Sensitivity Function

35. …because that’s where the
Visible Light
Why do we see light of these wavelengths?
…because that’s where the
Sun radiates EM energy
© Stephen E. Palmer, 2002

36. The Physics of Light Any patch of light can be completely described
physically by its spectrum: the number of photons
(per time unit) at each wavelength nm.
© Stephen E. Palmer, 2002

37. The Physics of Light Some examples of the spectra of light sources
© Stephen E. Palmer, 2002

38. The Physics of Light
Some examples of the reflectance spectra of surfaces
Red
Yellow
Blue
Purple
% Photons Reflected
Wavelength (nm)
© Stephen E. Palmer, 2002

39. The Psychophysical Correspondence
There is no simple functional description for the perceived
color of all lights under all viewing conditions, but …...
A helpful constraint:
Consider only physical spectra with normal distributions
mean
area
variance
© Stephen E. Palmer, 2002

40. The Psychophysical Correspondence
Mean
Hue
# Photons
Wavelength
© Stephen E. Palmer, 2002

41. The Psychophysical Correspondence
Variance
Saturation
Wavelength
# Photons
© Stephen E. Palmer, 2002

42. The Psychophysical Correspondence
Area
Brightness
# Photons
Wavelength
© Stephen E. Palmer, 2002

43. Physiology of Color Vision
Three kinds of cones:
Why are M and L cones so close?
© Stephen E. Palmer, 2002

44. Retinal Processing
© Stephen E. Palmer, 2002

45. Single Cell Recording Microelectrode Electrical response
Amplifier
Electrical response
(action potentials) mV
© Stephen E. Palmer, 2002

46. Single Cell Recording
© Stephen E. Palmer, 2002

47. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

48. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

49. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

50. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

51. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

52. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

53. Retinal Receptive Fields
RF of On-center Off-surround cells
© Stephen E. Palmer, 2002

54. Retinal Receptive Fields
RF of Off-center On-surround cells
Surround
Center
© Stephen E. Palmer, 2002

55. Retinal Receptive Fields

56. Retinal Receptive Fields
Receptive field structure in bipolar cells
Light
© Stephen E. Palmer, 2002

57. Retinal Receptive Fields
Receptive field structure in bipolar cells
© Stephen E. Palmer, 2002

58. Visual Cortex Cortical Area V1 aka: Primary visual cortex
Striate cortex
Brodman’s area 17
© Stephen E. Palmer, 2002

59. Cortical Receptive Fields
Single-cell recording from visual cortex
David Hubel & Thorston Wiesel
© Stephen E. Palmer, 2002

60. Cortical Receptive Fields
Single-cell recording from visual cortex
© Stephen E. Palmer, 2002

61. Cortical Receptive Fields
Three classes of cells in V1
Simple cells
Complex cells
Hypercomplex cells
© Stephen E. Palmer, 2002

62. Cortical Receptive Fields
Simple Cells: “Line Detectors”
© Stephen E. Palmer, 2002

63. Cortical Receptive Fields
Simple Cells: “Edge Detectors”
© Stephen E. Palmer, 2002

64. Cortical Receptive Fields
Constructing a line detector
© Stephen E. Palmer, 2002

65. Cortical Receptive Fields
Complex Cells 0o
© Stephen E. Palmer, 2002

66. Cortical Receptive Fields
Complex Cells
60o
© Stephen E. Palmer, 2002

67. Cortical Receptive Fields
Complex Cells
90o
© Stephen E. Palmer, 2002

68. Cortical Receptive Fields
Complex Cells
120o
© Stephen E. Palmer, 2002

69. Cortical Receptive Fields
Constructing a Complex Cell
© Stephen E. Palmer, 2002

70. Cortical Receptive Fields
Hypercomplex Cells
© Stephen E. Palmer, 2002

71. Cortical Receptive Fields
Hypercomplex Cells
© Stephen E. Palmer, 2002

72. Cortical Receptive Fields
Hypercomplex Cells
© Stephen E. Palmer, 2002

73. Cortical Receptive Fields
Hypercomplex Cells
“End-stopped” Cells
© Stephen E. Palmer, 2002

74. Cortical Receptive Fields
“End-stopped” Simple Cells
© Stephen E. Palmer, 2002

75. Cortical Receptive Fields
Constructing a Hypercomplex Cell
© Stephen E. Palmer, 2002

76. Mapping from Retina to V1

77. Why edges?

78. Why edges?