Computational Theories & Low-level Pixels To Percepts A. Efros, CMU, Spring 2009

Four Stages of Visual Perception © Stephen E. Palmer, 2002 Ceramic cup on a table David Marr, 1982

Four Stages of Visual Perception © Stephen E. Palmer, 2002 The Retinal Image An Image (blowup) Receptor Output

Four Stages of Visual Perception © Stephen E. Palmer, 2002 Image-based Representation Primal Sketch (Marr) An Image (Line Drawing) Retinal Image Image- based processes Edges Lines Blobs etc.

We likely throw away a lot

line drawings are universal

Four Stages of Visual Perception © Stephen E. Palmer, 2002 Surface-based Representation Primal Sketch 2.5-D Sketch Image-based Representation Surface- based processes Stereo Shading Motion etc.

Single Surface (Koenderink’s trick)

Four Stages of Visual Perception © Stephen E. Palmer, 2002 Surface-based Representation Primal Sketch 2.5-D Sketch Image-based Representation Surface- based processes Stereo Shading Motion etc.

Figure/Ground Organization  A contour belongs to one of the two (but not both) abutting regions. Figure (face) Ground (shapeless) Figure (Goblet) Ground (Shapeless) Important for the perception of shape

© Stephen E. Palmer, 2002 Properties of figures vs. grounds FigureGround Thing-likeNot thing-like CloserFarther ShapedExtends behind Figure-Ground Organization

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Surroundedness Figure-Ground Organization Surrounded region --> Figure Surrounding region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Size Figure-Ground Organization Smaller region --> Figure Larger region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Orientation Figure-Ground Organization Horizontal/vertical region --> Figure Oblique region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Contrast Figure-Ground Organization Higher contrast region --> Figure Lower contrast region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Symmetry Figure-Ground Organization Symmetrical region --> Figure Asymmetrical region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Convexity Figure-Ground Organization More convex region --> Figure Less convex region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Parallelism Figure-Ground Organization More parallel region --> Figure Less parallel region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Lower region Figure-Ground Organization Lower region --> Figure Upper region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Meaningfulness Figure-Ground Organization More meaningful region --> Figure Less meaningful region --> Ground

© Stephen E. Palmer, 2002 Relation to Depth Factors Figure-Ground Organization Figure-ground organization as edge assignment: To which side does the edge belong? Depth cues can also be figure-ground factors and Figure-ground factors can be depth cues. To the closer side. This fact connects figure-ground organization with depth perception.

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Occlusion Figure-Ground Organization Occluding region --> Figure Occluded region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Cast Shadows Figure-Ground Organization Shadowing region --> Figure Shadowed region --> Ground

© Stephen E. Palmer, 2002 Principles of figure-ground organization: Shading Figure-Ground Organization Shaded region --> Figure Nonshaded region --> Ground

Line Labeling > : contour direction + : convex edge - : concave edge possible junctions (constraints) Constraint Propagation [Clowes 1971, Huffman 1971; Waltz 1972; Malik 1986]

26

Line Labeling

Four Stages of Visual Perception © Stephen E. Palmer, 2002 Object-based Representation Object- based processes Grouping Parsing Completion etc. Surface-based Representation 2.5-D Sketch Volumetric Sketch

Geons (Biederman '87)

Four Stages of Visual Perception © Stephen E. Palmer, 2002 Category-based Representation Category- based processes Pattern- Recognition Spatial- description Object-based Representation Volumetric Sketch Basic-level Category Category: cup Color: light-gray Size: 6” Location: table

We likely throw away a lot

line drawings are universal

However, things are not so simple… ● Problems with feed-forward model of processing…

Junctions in Real Images

Are Junctions local evidence? J McDermott, 2004

© Stephen E. Palmer, Is grouping an early or late process? Early vs. Late Grouping ????

© Stephen E. Palmer, Before or after stereoscopic depth? (Rock & Brosgole, 1964) Early vs. Late Grouping

© Stephen E. Palmer, Before or after lightness constancy? (Rock, Nijhawan, Palmer & Tudor, 1992) Early vs. Late Grouping Opaque paper strip

© Stephen E. Palmer, Before or after visual completion? (Palmer, Neff & Beck, 1996) Early vs. Late Grouping

© Stephen E. Palmer, Before or after illusory contours? (Palmer & Nelson, 2000) ? Early vs. Late Grouping

© Stephen E. Palmer, Conclusion: Grouping can occur “late” Question: Can grouping also occur “early” (Palmer & Brooks, in preparation) Early vs. Late Grouping

© Stephen E. Palmer, Grouping affects shape constancy (Palmer & Brooks, in preparation) Ambiguous Flat oval Circle in depth Early vs. Late Grouping

© Stephen E. Palmer, Proximity effects Biased toward oval Biased toward circle Early vs. Late Grouping

© Stephen E. Palmer, Color similarity effects Biased toward ovalBiased toward circle Early vs. Late Grouping

© Stephen E. Palmer, Common fate effects Biased toward ovalBiased toward circle Early vs. Late Grouping

© Stephen E. Palmer, Conclusion: Grouping occurs both “early” and “late” -- possibly everywhere! Grouping Early vs. Late Grouping

two-tone images

hair (not shadow!) inferred external contours “attached shadow” contour “cast shadow” contour

Finding 3D structure in two-tone images requires distinguishing cast shadows, attached shadows, and areas of low reflectivity The images do not contain this information a priori (at low level) Cavanagh's argument

A Classical View of Vision Grouping / Segmentation Figure/Ground Organization Object and Scene Recognition pixels, features, edges, etc. Low-level Mid-level High-level

A Contemporary View of Vision Figure/Ground Organization Grouping / Segmentation Object and Scene Recognition pixels, features, edges, etc. Low-level Mid-level High-level But where we draw this line?

Question #1: What (if anything) should be done at the “Low-Level”? N.B. I have already told you everything that is known. From now on, there aren’t any answers.. Only questions…

Who cares? Why not just use pixels? Pixel differences vs. Perceptual differences

Eye is not a photometer! "Every light is a shade, compared to the higher lights, till you come to the sun; and every shade is a light, compared to the deeper shades, till you come to the night." — John Ruskin, 1879

Cornsweet Illusion

Campbell-Robson contrast sensitivity curve Sine wave

Metamers

Question #1: What (if anything) should be done at the “Low-Level”? i.e. What input stimulus should we be invariant to?

Invariant to: Brightness / Color changes? small brightness / color changes low-frequency changes But one can be too invariant

Invariant to: Edge contrast / reversal? I shouldn’t care what background I am on! but be careful of exaggerating noise

Representation choices Raw Pixels Gradients: Gradient Magnitude: Thresholded gradients (edge + sign): Thresholded gradient mag. (edges):

Spatial invariance Rotation, Translation, Scale Yes, but not too much… In brain: complex cells – partial invariance In Comp. Vision: histogram-binning methods (SIFT, GIST, Shape Context, etc) or, equivalently, blurring (e.g. Geometric Blur) -- will discuss later

Many lives of a boundary

Often, context-dependent… inputcanny human Maybe low-level is never enough?

1/f amplitude spectra for natural images (Field 1987) There are statistical regularities in the natural world, and image statistics reflect that. (Burton & Moorehead 1987; Field 1987; Tolhurst et al. 1992)

Why 1/f? Scale invariance Edges have 1/f structure Object distribution in real world (Ruderman 1997; Lee & Mumford 1999) (Image source: smokiesguidebook.com Slide content: Simoncelli & Olshausen 2001)

A closer look at amplitude spectra (Torralba & Oliva 2003)

Do natural image statistics matter? Sensory coding might exploit statistical regularities of our world according to various criteria: Representational efficiency Decorrelate input responses, make them independent, sparse, information theoretic metrics etc. Metabolic efficiency Spike efficiency, minimal wiring. Learning efficiency Sparseness, invariance, over completeness etc. Lots and lots of work; see reviews Graham & Field (2007), Simoncelli & Olshausen (2001)