1. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Lecture 38 of 42 Wednesday, 03 December 2008 William H. Hsu Department of Computing and Information Sciences, KSU KSOL course page: http://snipurl.com/v9v3http://snipurl.com/v9v3 Course web site: http://www.kddresearch.org/Courses/Fall-2008/CIS730http://www.kddresearch.org/Courses/Fall-2008/CIS730 Instructor home page: http://www.cis.ksu.edu/~bhsuhttp://www.cis.ksu.edu/~bhsu Reading for Next Class: Sections 22.1, 22.6-7, Russell & Norvig 2 nd edition Vision, Part 2 of 2 Discussion: Machine Problem 7

2. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Vision Outline Physiology of Vision (1 lecture) Overview of Human Visual Percetion (1 lecture)  Need presenter for Monday! Part I: Low-level vision (images as texture)  Texture segmentation, image retrieval, scene models, “Bag of words” representations Part II: Mid-level vision (segmentation)  Principles of grouping, Normalized Cuts, Mean-shift, DD-MCMC, Graph-cut, super-pixels Part III: 2D Recognition  Window scanning (Schniderman+Kanade, Viola+Jones)  Correspondence Matching (schanfer matching, housedorf distance, shape contexts, invariant features, active appearance models)  Recognition with Segmentation (top-down + buttom-up)  Words and Pictures http://www.cs.cmu.edu/~efros/courses/AP06/

3. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence So what do humans care about? slide by Fei Fei, Fergus & Torralba

4. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Verification: is that a bus? slide by Fei Fei, Fergus & Torralba

5. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Detection: are there cars? slide by Fei Fei, Fergus & Torralba

6. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Identification: is that a picture of Mao? slide by Fei Fei, Fergus & Torralba

7. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Object categorization sky building flag wall banner bus cars bus face street lamp slide by Fei Fei, Fergus & Torralba

8. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Scene and context categorization outdoor city traffic … slide by Fei Fei, Fergus & Torralba

9. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Rough 3D layout, depth ordering

10. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 1: view point variation Michelangelo 1475-1564

11. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 2: illumination slide credit: S. Ullman

12. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 3: occlusion Magritte, 1957

13. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 4: scale slide by Fei Fei, Fergus & Torralba

14. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 5: deformation Xu, Beihong 1943

15. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 6: background clutter Klimt, 1913

16. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 7: object intra-class variation slide by Fei-Fei, Fergus & Torralba

17. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 8: local ambiguity slide by Fei-Fei, Fergus & Torralba

18. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 9: the world behind the image

19. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence In this course, we will: Take a few baby steps…

20. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Physiology of Vision: a swift overview 16-721: Learning-Based Methods in Vision A. Efros, CMU, Spring 2007 Some figures from Steve Palmer

21. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Class Introductions Name: Research area / project / advisor What you want to learn in this class? When I am not working, I ______________ Favorite fruit:

22. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Image Formation Digital Camera The Eye Film

23. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Monocular Visual Field: 160 deg (w) X 135 deg (h) Binocular Visual Field: 200 deg (w) X 135 deg (h)

24. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Figures © Stephen E. Palmer, 2002 What do we see? 3D world2D image

25. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence What do we see? 3D world2D image Painted backdrop

26. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence The Plenoptic Function Q: What is the set of all things that we can ever see? A: The Plenoptic Function (Adelson & Bergen) Let’s start with a stationary person and try to parameterize everything that he can see… Figure by Leonard McMillan

27. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Grayscale snapshot is intensity of light  Seen from a single view point  At a single time  Averaged over the wavelengths of the visible spectrum (can also do P(x,y), but spherical coordinate are nicer) P(  )

28. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Color snapshot is intensity of light  Seen from a single view point  At a single time  As a function of wavelength P(  )

29. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Spherical Panorama All light rays through a point form a ponorama Totally captured in a 2D array -- P(  ) Where is the geometry??? See also: 2003 New Years Eve http://www.panoramas.dk/fullscreen3/f1.html

30. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence A movie is intensity of light  Seen from a single view point  Over time  As a function of wavelength P( ,t)

31. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Space-time images x y t

32. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Holographic movie is intensity of light  Seen from ANY viewpoint  Over time  As a function of wavelength P( ,t,V X,V Y,V Z )

33. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence The Plenoptic Function  Can reconstruct every possible view, at every moment, from every position, at every wavelength  Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality! Not bad for a function… P( ,t,V X,V Y,V Z )

34. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence The Eye is a camera The human eye is a camera!  Iris - colored annulus with radial muscles  Pupil - the hole (aperture) whose size is controlled by the iris  What’s the “film”?  photoreceptor cells (rods and cones) in the retina

35. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence The Retina

36. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Retina up-close Light

37. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence © Stephen E. Palmer, 2002 Cones cone-shaped less sensitive operate in high light color vision Two types of light-sensitive receptors Rods rod-shaped highly sensitive operate at night gray-scale vision

38. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Rod / Cone sensitivity The famous sock-matching problem…

39. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence © Stephen E. Palmer, 2002 Distribution of Rods and Cones Night Sky: why are there more stars off-center?

40. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Electromagnetic Spectrum http://www.yorku.ca/eye/photopik.htm Human Luminance Sensitivity Function

41. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Why do we see light of these wavelengths? © Stephen E. Palmer, 2002 …because that’s where the Sun radiates EM energy Visible Light

42. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Retinal Processing © Stephen E. Palmer, 2002

43. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Single Cell Recording Microelectrode Amplifier Electrical response (action potentials) mV © Stephen E. Palmer, 2002

44. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Single Cell Recording © Stephen E. Palmer, 2002

45. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Retinal Receptive Fields Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response © Stephen E. Palmer, 2002

46. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

47. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

48. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

49. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

50. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

51. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence RF of On-center Off-surround cells Retinal Receptive Fields © Stephen E. Palmer, 2002

52. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence RF of Off-center On-surround cells Retinal Receptive Fields © Stephen E. Palmer, 2002 Surround Center

53. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Retinal Receptive Fields

54. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in bipolar cells Light Retinal Receptive Fields © Stephen E. Palmer, 2002

55. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in bipolar cells Retinal Receptive Fields © Stephen E. Palmer, 2002

56. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence © Stephen E. Palmer, 2002 Visual Cortex aka: Primary visual cortex Striate cortex Brodman’s area 17 Cortical Area V1

57. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Single-cell recording from visual cortex David Hubel & Thorston Wiesel © Stephen E. Palmer, 2002

58. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Single-cell recording from visual cortex © Stephen E. Palmer, 2002

59. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Three classes of cells in V1 Simple cells Complex cells Hypercomplex cells © Stephen E. Palmer, 2002

60. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Simple Cells: “Line Detectors” © Stephen E. Palmer, 2002

61. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Simple Cells: “Edge Detectors” © Stephen E. Palmer, 2002

62. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Constructing a line detector © Stephen E. Palmer, 2002

63. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Complex Cells 0o0o © Stephen E. Palmer, 2002

64. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Complex Cells 60 o © Stephen E. Palmer, 2002

65. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Complex Cells 90 o © Stephen E. Palmer, 2002

66. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Complex Cells 120 o © Stephen E. Palmer, 2002

67. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Constructing a Complex Cell © Stephen E. Palmer, 2002

68. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

69. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

70. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

71. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Hypercomplex Cells “End-stopped” Cells © Stephen E. Palmer, 2002

72. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields “End-stopped” Simple Cells © Stephen E. Palmer, 2002

73. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Constructing a Hypercomplex Cell © Stephen E. Palmer, 2002

74. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Mapping from Retina to V1

75. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Why edges? So, why “edge-like” structures in the Plenoptic Function?

76. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Because our world is structured!

77. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Problem: Dynamic Range 1500 1 1 25,000 400,000 2,000,000,000 The real world is High dynamic range

78. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence pixel (312, 284) = 42 Image 42 photos? Is Camera a photometer?

79. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Long Exposure 10 -6 10 6 10 -6 10 6 Real world Picture 0 to 255 High dynamic range

80. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Short Exposure 10 -6 10 6 10 -6 10 6 Real world Picture 0 to 255 High dynamic range

81. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Varying Exposure

82. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence What does the eye sees? The eye has a huge dynamic range Do we see a true radiance map?

83. Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Eye is not a photometer! "Every light is a shade, compared to the higher lights, till you come to the sun; and every shade is a light, compared to the deeper shades, till you come to the night." — John Ruskin, 1879