1. Physiology of Vision: a swift overview
16-721: Learning-Based Methods in Vision
A. Efros, CMU, Spring 2007
Some figures from Steve Palmer

2. Class Introductions Name: Research area / project / advisor
What you want to learn in this class?
When I am not working, I ______________
Favorite fruit:

3. Image Formation
Digital Camera
Film
The Eye

4. Monocular Visual Field: 160 deg (w) X 135 deg (h) Binocular Visual Field: 200 deg (w) X 135 deg (h)

5. What do we see?
3D world
2D image
Figures © Stephen E. Palmer, 2002

6. What do we see?
3D world
2D image
Painted
backdrop

7. The Plenoptic Function
Figure by Leonard McMillan
Q: What is the set of all things that we can ever see?
A: The Plenoptic Function (Adelson & Bergen)
Let’s start with a stationary person and try to parameterize everything that he can see…

8. Grayscale snapshot P(q,f) is intensity of light
Seen from a single view point
At a single time
Averaged over the wavelengths of the visible spectrum
(can also do P(x,y), but spherical coordinate are nicer)

9. Color snapshot P(q,f,l) is intensity of light
Seen from a single view point
At a single time
As a function of wavelength

10. Spherical Panorama All light rays through a point form a ponorama
See also: 2003 New Years Eve
All light rays through a point form a ponorama
Totally captured in a 2D array -- P(q,f)
Where is the geometry???

11. A movie P(q,f,l,t) is intensity of light Seen from a single view point
Over time
As a function of wavelength

12. Space-time images t y x

13. Holographic movie P(q,f,l,t,VX,VY,VZ) is intensity of light
Seen from ANY viewpoint
Over time
As a function of wavelength

14. The Plenoptic Function
P(q,f,l,t,VX,VY,VZ)
Can reconstruct every possible view, at every moment, from every position, at every wavelength
Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality! Not bad for a function…

15. The Eye is a camera The human eye is a camera!
Iris - colored annulus with radial muscles
Pupil - the hole (aperture) whose size is controlled by the iris
What’s the “film”?
photoreceptor cells (rods and cones) in the retina

16. The Retina

17. Retina up-close
Light

18. Two types of light-sensitive receptors
Cones
cone-shaped
less sensitive
operate in high light
color vision
Rods
rod-shaped
highly sensitive
operate at night
gray-scale vision
© Stephen E. Palmer, 2002

19. Rod / Cone sensitivity
The famous sock-matching problem…

20. Distribution of Rods and Cones
Night Sky: why are there more stars off-center?
© Stephen E. Palmer, 2002

21. Electromagnetic Spectrum
At least 3 spectral bands required (e.g. R,G,B)
Human Luminance Sensitivity Function

22. …because that’s where the
Visible Light
Why do we see light of these wavelengths?
…because that’s where the
Sun radiates EM energy
© Stephen E. Palmer, 2002

23. The Physics of Light Any patch of light can be completely described
physically by its spectrum: the number of photons
(per time unit) at each wavelength nm.
© Stephen E. Palmer, 2002

24. The Physics of Light Some examples of the spectra of light sources
© Stephen E. Palmer, 2002

25. The Physics of Light
Some examples of the reflectance spectra of surfaces
Red
Yellow
Blue
Purple
% Photons Reflected
Wavelength (nm)
© Stephen E. Palmer, 2002

26. The Psychophysical Correspondence
There is no simple functional description for the perceived
color of all lights under all viewing conditions, but …...
A helpful constraint:
Consider only physical spectra with normal distributions
mean
area
variance
© Stephen E. Palmer, 2002

27. The Psychophysical Correspondence
Mean
Hue
# Photons
Wavelength
© Stephen E. Palmer, 2002

28. The Psychophysical Correspondence
Variance
Saturation
Wavelength
# Photons
© Stephen E. Palmer, 2002

29. The Psychophysical Correspondence
Area
Brightness
# Photons
Wavelength
© Stephen E. Palmer, 2002

30. Physiology of Color Vision
Three kinds of cones:
Why are M and L cones so close?
© Stephen E. Palmer, 2002

31. Retinal Processing
© Stephen E. Palmer, 2002

32. Single Cell Recording Microelectrode Electrical response
Amplifier
Electrical response
(action potentials) mV
© Stephen E. Palmer, 2002

33. Single Cell Recording
© Stephen E. Palmer, 2002

34. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

35. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

36. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

37. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

38. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

39. Retinal Receptive Fields
Receptive field structure in ganglion cells:
On-center Off-surround
Stimulus condition
Electrical response
© Stephen E. Palmer, 2002

40. Retinal Receptive Fields
RF of On-center Off-surround cells
© Stephen E. Palmer, 2002

41. Retinal Receptive Fields
RF of Off-center On-surround cells
Surround
Center
© Stephen E. Palmer, 2002

42. Retinal Receptive Fields

43. Retinal Receptive Fields
Receptive field structure in bipolar cells
Light
© Stephen E. Palmer, 2002

44. Retinal Receptive Fields
Receptive field structure in bipolar cells
© Stephen E. Palmer, 2002

45. Visual Cortex Cortical Area V1 aka: Primary visual cortex
Striate cortex
Brodman’s area 17
© Stephen E. Palmer, 2002

46. Cortical Receptive Fields
Single-cell recording from visual cortex
David Hubel & Thorston Wiesel
© Stephen E. Palmer, 2002

47. Cortical Receptive Fields
Single-cell recording from visual cortex
© Stephen E. Palmer, 2002

48. Cortical Receptive Fields
Three classes of cells in V1
Simple cells
Complex cells
Hypercomplex cells
© Stephen E. Palmer, 2002

49. Cortical Receptive Fields
Simple Cells: “Line Detectors”
© Stephen E. Palmer, 2002

50. Cortical Receptive Fields
Simple Cells: “Edge Detectors”
© Stephen E. Palmer, 2002

51. Cortical Receptive Fields
Constructing a line detector
© Stephen E. Palmer, 2002

52. Cortical Receptive Fields
Complex Cells 0o
© Stephen E. Palmer, 2002

53. Cortical Receptive Fields
Complex Cells
60o
© Stephen E. Palmer, 2002

54. Cortical Receptive Fields
Complex Cells
90o
© Stephen E. Palmer, 2002

55. Cortical Receptive Fields
Complex Cells
120o
© Stephen E. Palmer, 2002

56. Cortical Receptive Fields
Constructing a Complex Cell
© Stephen E. Palmer, 2002

57. Cortical Receptive Fields
Hypercomplex Cells
© Stephen E. Palmer, 2002

58. Cortical Receptive Fields
Hypercomplex Cells
© Stephen E. Palmer, 2002

59. Cortical Receptive Fields
Hypercomplex Cells
© Stephen E. Palmer, 2002

60. Cortical Receptive Fields
Hypercomplex Cells
“End-stopped” Cells
© Stephen E. Palmer, 2002

61. Cortical Receptive Fields
“End-stopped” Simple Cells
© Stephen E. Palmer, 2002

62. Cortical Receptive Fields
Constructing a Hypercomplex Cell
© Stephen E. Palmer, 2002

63. Mapping from Retina to V1

64. Why edges?
So, why “edge-like” structures in the Plenoptic Function?

65. Because our world is structured!

66. Problem: Dynamic Range
The real world is
High dynamic range 1
1500
25,000
400,000
2,000,000,000

67. Is Camera a photometer?
Image
pixel (312, 284) = 42
42 photos? 2

68. Long Exposure 10-6 High dynamic range 106 10-6 106 0 to 255 Real world
Picture
0 to 255

69. Short Exposure 10-6 High dynamic range 106 10-6 106 0 to 255
Real world
10-6
106
Picture
0 to 255

70. Varying Exposure

71. What does the eye sees? The eye has a huge dynamic range
Do we see a true radiance map?

72. Eye is not a photometer!
"Every light is a shade, compared to the higher lights, till you come to the sun; and every shade is a light, compared to the deeper shades, till you come to the night."
— John Ruskin, 1879

73. Cornsweet Illusion

74. Campbell-Robson contrast sensitivity curve
Sine wave
Campbell-Robson contrast sensitivity curve

75. Metamers
Eye is sensitive to changes
(more on this later…)