1. Physiology of Vision: a swift overview 16-721: Advanced Machine Perception A. Efros, CMU, Spring 2006 Some figures from Steve Palmer

2. Class Introductions Name: Research area / project / advisor What you want to learn in this class? When I am not working, I ______________ Favorite fruit: Class mailing list…

3. Image Formation Digital Camera The Eye Film

4. Monocular Visual Field: 160 deg (w) X 135 deg (h) Binocular Visual Field: 200 deg (w) X 135 deg (h)

5. Figures © Stephen E. Palmer, 2002 What do we see? 3D world2D image

6. What do we see? 3D world2D image Painted backdrop

7. The Plenoptic Function Q: What is the set of all things that we can ever see? A: The Plenoptic Function (Adelson & Bergen) Let’s start with a stationary person and try to parameterize everything that he can see… Figure by Leonard McMillan

8. Grayscale snapshot is intensity of light Seen from a single view point At a single time Averaged over the wavelengths of the visible spectrum (can also do P(x,y), but spherical coordinate are nicer) P(  )

9. Color snapshot is intensity of light Seen from a single view point At a single time As a function of wavelength P(  )

10. Spherical Panorama All light rays through a point form a ponorama Totally captured in a 2D array -- P(  ) Where is the geometry??? See also: 2003 New Years Eve http://www.panoramas.dk/fullscreen3/f1.html

11. A movie is intensity of light Seen from a single view point Over time As a function of wavelength P( ,t)

12. Space-time images x y t

13. Holographic movie is intensity of light Seen from ANY viewpoint Over time As a function of wavelength P( ,t,V X,V Y,V Z )

14. The Plenoptic Function Can reconstruct every possible view, at every moment, from every position, at every wavelength Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality! Not bad for a function… P( ,t,V X,V Y,V Z )

15. The Eye is a camera The human eye is a camera! Iris - colored annulus with radial muscles Pupil - the hole (aperture) whose size is controlled by the iris What’s the “film”? –photoreceptor cells (rods and cones) in the retina

16. The Retina

17. Retina up-close Light

18. © Stephen E. Palmer, 2002 Cones cone-shaped less sensitive operate in high light color vision Two types of light-sensitive receptors Rods rod-shaped highly sensitive operate at night gray-scale vision

19. Rod / Cone sensitivity The famous sock-matching problem…

20. © Stephen E. Palmer, 2002 Distribution of Rods and Cones Night Sky: why are there more stars off-center?

21. Electromagnetic Spectrum http://www.yorku.ca/eye/photopik.htm Human Luminance Sensitivity Function

22. Why do we see light of these wavelengths? © Stephen E. Palmer, 2002 …because that’s where the Sun radiates EM energy Visible Light

23. The Physics of Light Any patch of light can be completely described physically by its spectrum: the number of photons (per time unit) at each wavelength 400 - 700 nm. © Stephen E. Palmer, 2002

24. The Physics of Light Some examples of the spectra of light sources © Stephen E. Palmer, 2002

25. The Physics of Light Some examples of the reflectance spectra of surfaces Wavelength (nm) % Photons Reflected Red 400 700 Yellow 400 700 Blue 400 700 Purple 400 700 © Stephen E. Palmer, 2002

26. The Psychophysical Correspondence There is no simple functional description for the perceived color of all lights under all viewing conditions, but …... A helpful constraint: Consider only physical spectra with normal distributions area mean variance © Stephen E. Palmer, 2002

27. The Psychophysical Correspondence MeanHue # Photons Wavelength © Stephen E. Palmer, 2002

28. The Psychophysical Correspondence VarianceSaturation Wavelength # Photons © Stephen E. Palmer, 2002

29. The Psychophysical Correspondence AreaBrightness # Photons Wavelength © Stephen E. Palmer, 2002

30. Three kinds of cones: Physiology of Color Vision Why are M and L cones so close?

31. Retinal Processing © Stephen E. Palmer, 2002

32. Single Cell Recording Microelectrode Amplifier Electrical response (action potentials) mV © Stephen E. Palmer, 2002

33. Single Cell Recording © Stephen E. Palmer, 2002

34. Retinal Receptive Fields Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response © Stephen E. Palmer, 2002

35. Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

36. Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

37. Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

38. Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

39. Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

40. RF of On-center Off-surround cells Retinal Receptive Fields © Stephen E. Palmer, 2002

41. RF of Off-center On-surround cells Retinal Receptive Fields © Stephen E. Palmer, 2002 Surround Center

42. Retinal Receptive Fields

43. Receptive field structure in bipolar cells Light Retinal Receptive Fields © Stephen E. Palmer, 2002

44. Receptive field structure in bipolar cells Retinal Receptive Fields © Stephen E. Palmer, 2002

45. Visual Cortex aka: Primary visual cortex Striate cortex Brodman’s area 17 Cortical Area V1

46. Cortical Receptive Fields Single-cell recording from visual cortex David Hubel & Thorston Wiesel © Stephen E. Palmer, 2002

47. Cortical Receptive Fields Single-cell recording from visual cortex © Stephen E. Palmer, 2002

48. Cortical Receptive Fields Three classes of cells in V1 Simple cells Complex cells Hypercomplex cells © Stephen E. Palmer, 2002

49. Cortical Receptive Fields Simple Cells: “Line Detectors” © Stephen E. Palmer, 2002

50. Cortical Receptive Fields Simple Cells: “Edge Detectors” © Stephen E. Palmer, 2002

51. Cortical Receptive Fields Constructing a line detector © Stephen E. Palmer, 2002

52. Cortical Receptive Fields Complex Cells 0o0o © Stephen E. Palmer, 2002

53. Cortical Receptive Fields Complex Cells 60 o © Stephen E. Palmer, 2002

54. Cortical Receptive Fields Complex Cells 90 o © Stephen E. Palmer, 2002

55. Cortical Receptive Fields Complex Cells 120 o © Stephen E. Palmer, 2002

56. Cortical Receptive Fields Constructing a Complex Cell © Stephen E. Palmer, 2002

57. Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

58. Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

59. Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

60. Cortical Receptive Fields Hypercomplex Cells “End-stopped” Cells © Stephen E. Palmer, 2002

61. Cortical Receptive Fields “End-stopped” Simple Cells © Stephen E. Palmer, 2002

62. Cortical Receptive Fields Constructing a Hypercomplex Cell © Stephen E. Palmer, 2002

63. Mapping from Retina to V1

64. Why edges? So, why “edge-like” structures in the Plenoptic Function?

65. Because our world is structured!

66. Problem: Dynamic Range 1500 1 1 25,000 400,000 2,000,000,000 The real world is High dynamic range

67. pixel (312, 284) = 42 Image 42 photos? Is Camera a photometer?

68. Long Exposure 10 -6 10 6 10 -6 10 6 Real world Picture 0 to 255 High dynamic range

69. Short Exposure 10 -6 10 6 10 -6 10 6 Real world Picture 0 to 255 High dynamic range

70. Varying Exposure

71. What does the eye sees? The eye has a huge dynamic range Do we see a true radiance map?

72. Eye is not a photometer! "Every light is a shade, compared to the higher lights, till you come to the sun; and every shade is a light, compared to the deeper shades, till you come to the night." — John Ruskin, 1879

73. Cornsweet Illusion

74. Campbell-Robson contrast sensitivity curve Sine wave

75. Metamers Eye is sensitive to changes (more on this later…)