Computational Vision Jitendra Malik University of California, Berkeley

What is in an image? The input is just an array of brightness values; humans perceive structure in it.

From Pixels to Perception Tiger Grass Water Sand outdoor wildlife Tiger tail eye legs head back shadow mouse

If visual processing was purely feedforward…(it isn’t) Pixels Local Neighborhoods Contours Surfaces Tiger Grass Water Sand Objects Scenes Low-level Image Processing Mid-level Grouping Figure/Ground Surface Attributes High-level Recognition

Boundaries of image regions defined by a number of attributes  Brightness/color  Texture  Motion  Binocular disparity  Familiar configuration

Grouping is hierarchical A BC A,C are refinements of B A,C are mutual refinements A,B,C represent the same percept Image BGL-birdR-bird grass bush head eye beak far body head eye beak body Perceptual organization forms a tree: Two segmentations are consistent when they can be explained by the same segmentation tree

Humans assign a depth ordering to surfaces across a contour  R1 appears in front of R2  R2 appears in front of R3 This can be done for images of natural scenes …

Figure-Ground Labeling - - red is near; blue is far

Figure/Ground Organization  A contour belongs to one of the two (but not both) abutting regions. Figure (face) Ground (shapeless) Figure (Goblet) Ground (Shapeless) Important for the perception of shape

Some other aspects of perceptual organization Good continuation Amodal completion Modal completion

What do we see here?

And here?

Some Pictorial Cues

Support, Size ? ? ? 1 3 2

Cast Shadows

Shading

Measuring Surface Orientation

Binocular Stereopsis

Optical flow for a pilot

Object Category Recognition

Shape variation within a category  D’Arcy Thompson: On Growth and Form, 1917  studied transformations between shapes of organisms

Attneave’s Cat (1954) Line drawings convey most of the information

Objects are in Scenes

Human stick figure from single image Input imageStick figureSupport masks

This is hard…  Variety of poses  Clothing  Missing parts  Small support for parts  Background clutter

Taxonomy and Partonomy  Taxonomy: E.g. Cats are in the order Felidae which in turn is in the class Mammalia  Recognition can be at multiple levels of categorization, or be identification at the level of specific individuals, as in faces.  Partonomy: Objects have parts, they have subparts and so on. The human body contains the head, which in turn contains the eyes.  These notions apply equally well to scenes and to activities.  Psychologists have argued that there is a “ basic-level ” at which categorization is fastest (Eleanor Rosch et al).  In a partonomy each level contributes useful information for recognition.

Visual Control of Action  Locomotion  Navigation/Way-finding  Obstacle Avoidance  Manipulation  Grasping  Pick and Place  Tool use

Camera Obscura (Reinerus Gemma-Frisius, 1544)

Camera Obscura (Angelo Sala, )