Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Lecture 38 of 42 Wednesday, 03 December 2008 William H. Hsu Department of Computing and Information Sciences, KSU KSOL course page: Course web site: Instructor home page: Reading for Next Class: Sections 22.1, , Russell & Norvig 2 nd edition Vision, Part 2 of 2 Discussion: Machine Problem 7

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Vision Outline Physiology of Vision (1 lecture) Overview of Human Visual Percetion (1 lecture)  Need presenter for Monday! Part I: Low-level vision (images as texture)  Texture segmentation, image retrieval, scene models, “Bag of words” representations Part II: Mid-level vision (segmentation)  Principles of grouping, Normalized Cuts, Mean-shift, DD-MCMC, Graph-cut, super-pixels Part III: 2D Recognition  Window scanning (Schniderman+Kanade, Viola+Jones)  Correspondence Matching (schanfer matching, housedorf distance, shape contexts, invariant features, active appearance models)  Recognition with Segmentation (top-down + buttom-up)  Words and Pictures

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence So what do humans care about? slide by Fei Fei, Fergus & Torralba

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Verification: is that a bus? slide by Fei Fei, Fergus & Torralba

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Detection: are there cars? slide by Fei Fei, Fergus & Torralba

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Identification: is that a picture of Mao? slide by Fei Fei, Fergus & Torralba

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Object categorization sky building flag wall banner bus cars bus face street lamp slide by Fei Fei, Fergus & Torralba

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Scene and context categorization outdoor city traffic … slide by Fei Fei, Fergus & Torralba

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Rough 3D layout, depth ordering

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 1: view point variation Michelangelo

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 2: illumination slide credit: S. Ullman

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 3: occlusion Magritte, 1957

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 4: scale slide by Fei Fei, Fergus & Torralba

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 5: deformation Xu, Beihong 1943

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 6: background clutter Klimt, 1913

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 7: object intra-class variation slide by Fei-Fei, Fergus & Torralba

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 8: local ambiguity slide by Fei-Fei, Fergus & Torralba

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Challenges 9: the world behind the image

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence In this course, we will: Take a few baby steps…

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Physiology of Vision: a swift overview : Learning-Based Methods in Vision A. Efros, CMU, Spring 2007 Some figures from Steve Palmer

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Class Introductions Name: Research area / project / advisor What you want to learn in this class? When I am not working, I ______________ Favorite fruit:

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Image Formation Digital Camera The Eye Film

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Monocular Visual Field: 160 deg (w) X 135 deg (h) Binocular Visual Field: 200 deg (w) X 135 deg (h)

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Figures © Stephen E. Palmer, 2002 What do we see? 3D world2D image

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence What do we see? 3D world2D image Painted backdrop

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence The Plenoptic Function Q: What is the set of all things that we can ever see? A: The Plenoptic Function (Adelson & Bergen) Let’s start with a stationary person and try to parameterize everything that he can see… Figure by Leonard McMillan

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Grayscale snapshot is intensity of light  Seen from a single view point  At a single time  Averaged over the wavelengths of the visible spectrum (can also do P(x,y), but spherical coordinate are nicer) P(  )

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Color snapshot is intensity of light  Seen from a single view point  At a single time  As a function of wavelength P(  )

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Spherical Panorama All light rays through a point form a ponorama Totally captured in a 2D array -- P(  ) Where is the geometry??? See also: 2003 New Years Eve

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence A movie is intensity of light  Seen from a single view point  Over time  As a function of wavelength P( ,t)

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Space-time images x y t

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Holographic movie is intensity of light  Seen from ANY viewpoint  Over time  As a function of wavelength P( ,t,V X,V Y,V Z )

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence The Plenoptic Function  Can reconstruct every possible view, at every moment, from every position, at every wavelength  Contains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality! Not bad for a function… P( ,t,V X,V Y,V Z )

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence The Eye is a camera The human eye is a camera!  Iris - colored annulus with radial muscles  Pupil - the hole (aperture) whose size is controlled by the iris  What’s the “film”?  photoreceptor cells (rods and cones) in the retina

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence The Retina

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Retina up-close Light

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence © Stephen E. Palmer, 2002 Cones cone-shaped less sensitive operate in high light color vision Two types of light-sensitive receptors Rods rod-shaped highly sensitive operate at night gray-scale vision

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Rod / Cone sensitivity The famous sock-matching problem…

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence © Stephen E. Palmer, 2002 Distribution of Rods and Cones Night Sky: why are there more stars off-center?

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Electromagnetic Spectrum Human Luminance Sensitivity Function

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Why do we see light of these wavelengths? © Stephen E. Palmer, 2002 …because that’s where the Sun radiates EM energy Visible Light

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Retinal Processing © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Single Cell Recording Microelectrode Amplifier Electrical response (action potentials) mV © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Single Cell Recording © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Retinal Receptive Fields Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in ganglion cells: On-center Off-surround Stimulus condition Electrical response Retinal Receptive Fields © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence RF of On-center Off-surround cells Retinal Receptive Fields © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence RF of Off-center On-surround cells Retinal Receptive Fields © Stephen E. Palmer, 2002 Surround Center

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Retinal Receptive Fields

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in bipolar cells Light Retinal Receptive Fields © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Receptive field structure in bipolar cells Retinal Receptive Fields © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence © Stephen E. Palmer, 2002 Visual Cortex aka: Primary visual cortex Striate cortex Brodman’s area 17 Cortical Area V1

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Single-cell recording from visual cortex David Hubel & Thorston Wiesel © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Single-cell recording from visual cortex © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Three classes of cells in V1 Simple cells Complex cells Hypercomplex cells © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Simple Cells: “Line Detectors” © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Simple Cells: “Edge Detectors” © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Constructing a line detector © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Complex Cells 0o0o © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Complex Cells 60 o © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Complex Cells 90 o © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Complex Cells 120 o © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Constructing a Complex Cell © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Hypercomplex Cells © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Hypercomplex Cells “End-stopped” Cells © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields “End-stopped” Simple Cells © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Cortical Receptive Fields Constructing a Hypercomplex Cell © Stephen E. Palmer, 2002

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Mapping from Retina to V1

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Why edges? So, why “edge-like” structures in the Plenoptic Function?

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Because our world is structured!

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Problem: Dynamic Range , ,000 2,000,000,000 The real world is High dynamic range

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence pixel (312, 284) = 42 Image 42 photos? Is Camera a photometer?

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Long Exposure Real world Picture 0 to 255 High dynamic range

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Short Exposure Real world Picture 0 to 255 High dynamic range

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Varying Exposure

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence What does the eye sees? The eye has a huge dynamic range Do we see a true radiance map?

Computing & Information Sciences Kansas State University Wednesday, 03 Dec 2008CIS 530 / 730: Artificial Intelligence Eye is not a photometer! "Every light is a shade, compared to the higher lights, till you come to the sun; and every shade is a light, compared to the deeper shades, till you come to the night." — John Ruskin, 1879