What should be done at the Low Level? 16-721: Learning-Based Methods in Vision A. Efros, CMU, Spring 2009

Class Introductions Name: Research area / project / advisor What you want to learn in this class? When I am not working, I ______________ Favorite fruit:

Analysis Projects / Presentations Wed: Varun note-taker: Dan Next Wed: Dan note-taker: Edward Dan and Edward need to meet with me ASAP Varun needs to meet second time

Four Stages of Visual Perception Ceramic cup on a table David Marr, 1982 © Stephen E. Palmer, 2002

Four Stages of Visual Perception The Retinal Image An Image (blowup) Receptor Output © Stephen E. Palmer, 2002

Four Stages of Visual Perception Retinal Image Image-based Representation An Image Primal Sketch (Marr) Image- based processes Edges Lines Blobs etc. (Line Drawing) © Stephen E. Palmer, 2002

Four Stages of Visual Perception Image-based Representation Surface-based Representation Primal Sketch 2.5-D Sketch Surface- based processes Stereo Shading Motion etc. © Stephen E. Palmer, 2002

Koenderink’s trick

Four Stages of Visual Perception Surface-based Representation Object-based Representation 2.5-D Sketch Volumetric Sketch Object- based processes Grouping Parsing Completion etc. © Stephen E. Palmer, 2002

Geons (Biederman '87)

Four Stages of Visual Perception Object-based Representation Category-based Representation Volumetric Sketch Category- based processes Pattern- Recognition Spatial- description Basic-level Category Category: cup Color: light-gray Size: 6” Location: table © Stephen E. Palmer, 2002

We likely throw away a lot

line drawings are universal

However, things are not so simple… Problems with feed-forward model of processing…

two-tone images

“attached shadow” contour hair (not shadow!) “cast shadow” contour inferred external contours

Cavanagh's argument Finding 3D structure in two-tone images requires distinguishing cast shadows, attached shadows, and areas of low reflectivity The images do not contain this information a priori (at low level)

Feedforward vs. feedback models Marr's model (circa 1980) Cavanagh’s Model (circa 1990s) object recognition by matching 3D models Object basic recognition with 2D primitives memory 3D model 3D shape 2½D sketch 2D shape feedback primal sketch reconstruction of shape from image features stimulus stimulus

A Classical View of Vision Object and Scene Recognition High-level Figure/Ground Organization Mid-level Grouping / Segmentation In fact, here is a classical view of visual perception, in which figure/ground organization actually plays a very important role. In this linear architecture, first we have the image on the retina, then pixels in the image are grouped together into regions and contours. After grouping, figure/ground origination takes place, which assigns the ownership of contours and forms the perception of shape. Based on that, we have the recognition of objects and scenes at the end. Low-level pixels, features, edges, etc.

A Contemporary View of Vision Object and Scene Recognition High-level Figure/Ground Organization Grouping / Segmentation Mid-level In fact, here is a classical view of visual perception, in which figure/ground organization actually plays a very important role. In this linear architecture, first we have the image on the retina, then pixels in the image are grouped together into regions and contours. After grouping, figure/ground origination takes place, which assigns the ownership of contours and forms the perception of shape. Based on that, we have the recognition of objects and scenes at the end. But where we draw this line? Low-level pixels, features, edges, etc.

Question #1: What (if anything) should be done at the “Low-Level”? N.B. I have already told you everything that is known. From now on, there aren’t any answers.. Only questions…

Who cares? Why not just use pixels? Pixel differences vs. Perceptual differences

Eye is not a photometer! "Every light is a shade, compared to the higher lights, till you come to the sun; and every shade is a light, compared to the deeper shades, till you come to the night." — John Ruskin, 1879

Cornsweet Illusion

Campbell-Robson contrast sensitivity curve Sine wave Campbell-Robson contrast sensitivity curve

Metamers

Question #1: What (if anything) should be done at the “Low-Level”? i.e. What input stimulus should we be invariant to?

Invariant to: Brightness / Color changes? low-frequency changes small brightness / color changes But one can be too invariant

Invariant to: Edge contrast / reversal? I shouldn’t care what background I am on! but be careful of exaggerating noise

Representation choices Raw Pixels Gradients: Gradient Magnitude: Thresholded gradients (edge + sign): Thresholded gradient mag. (edges):

Typical filter bank

pyramid (e.g. wavelet, stearable, etc) Filters Input image

What does it capture? v = F * Patch (where F is filter matrix)

Why these filters?

Learned filters

Spatial invariance Rotation, Translation, Scale Yes, but not too much… In brain: complex cells – partial invariance In Comp. Vision: histogram-binning methods (SIFT, GIST, Shape Context, etc) or, equivalently, blurring (e.g. Geometric Blur) -- will discuss later

Many lives of a boundary

Often, context-dependent… input canny human Maybe low-level is never enough?