1Ellen L. Walker 3D Vision Why? The world is 3D Not all useful information is readily available in 2D Why so hard? “Inverse problem”: one image = many scenes Complex relationship between objects & pixels Noise, occlusion, etc. What can we do? Use "hints" from our knowledge of the world Add more information to the problem!

2Ellen L. Walker Labeling Image Edges Many edge labels carry 3D information Occluding blade (>) Occluding surface to right along arrow Convex crease (+) Edge is closer than both surfaces Concave crease (–) Edge is further than both surfaces Limb (>>) Edge is "horizon" of curving-away surface Others are about reflectance or illumination changes Mark (M) Change due to paint or material boundary Illumination Boundary (S) Shadow edge

3Ellen L. Walker Intrinsic Image Pixel Contents Depth (range) Orientation (surface normal) Illumination Albedo (reflectance) Given this information, the picture (intensity values) can be completely reconstructed

4Ellen L. Walker 3D Cues in Single 2D Images Occlusion Occluding objects are closer to the camera The crossbar of a T-junction belongs to the occluding object Perspective scaling and foreshortening If two copies of the same object appear in a picture, the smaller one is further away. Scaling is parallel to the image plane Foreshortening is perpendicular to the image plane Texture Gradient Since texture is composed of repeated patterns, changes in size and density of texture convey depth cues

5Ellen L. Walker "Shape-From" Methods Use a cue (e.g. texture, shading) from a small region of an image Cues generally give partial surface orientation information E.g. degree of tilt Related cues can give "boundary conditions" to start from Solve for continuous surfaces that satisfy both the general and boundary constraints

6Ellen L. Walker Example: Shape From Shading Figure 12.2

7Ellen L. Walker Gradient Space Represents Surface Normal

8Ellen L. Walker Reflectance Geometry Three directions are important: normal to the surface, surface to light source, and surface to camera

9Ellen L. Walker Types of Reflection (Review) Specular reflection (mirror) Color depends on light source color Limited scattering: angle of incidence = angle of reflection Camera sees light if it’s pointed in the right direction Nearly all light is reflected Lambertian reflection (matte) Color depends on material properties of object Light evenly scattered throughout half-space Camera sees light if surface is visible Amount of light reflected is proportional to angle between surface and light source

10Ellen L. Walker Shape From Shading Lambertian surface, known light source direction Reflective component (highlights) can be subtracted out in preprocessing Relative brightness of surface patches constrain their directions Darker patches are more tilted away A given brightness value represents a circle in gradient space Boundary pixels indicate surface at 90 degrees from normal (if smooth surface) Solve an optimization problem: brightness term and smoothness term

11Ellen L. Walker Shape From Shading (cont’d) Brightness term Intensity is a function of reflectance, and reflectance is a function of surface normals (p,q) and light source direction (v x, v y, v z ) Smoothness term Try to minimize integral of partial derivatives of p and q in x and y direction

12Ellen L. Walker Photometric Stereo Extension to multiple "images" Lambertian surface, several light sources Each image has one light source, constrains surfaces Solve an overdetermined linear (matrix) system - like camera calibration with extra points Implementation Surround your object with a frame containing inward- pointing lights Take an image with each light in turn Use images and known light directions to solve the equations.

13Ellen L. Walker More Variations SHINY Photometric stereo system for highly reflective materials Used to accurately characterize welds Accurate color determination (plastic objects) Separate highlights from matte portion Determine illumination color from highlight Determine object color Create “matte object” for photometric stereo or shape from shading

14Ellen L. Walker Shape From Texture Figure 12.3

15Ellen L. Walker Shape from Texture Transformation of original texture related to surface normal Solve for affine transformation between original texel and viewed texel Transformation depends on surface normal & distance (Assume camera is far enough to avoid worst perspective distortion) If the original texel is known, transformations can be computed directly If the original texel is unknown, assume the largest visible texel is directly facing the camera Use smoothness or shape constraints to eliminate alternatives

16Ellen L. Walker Shape from Focus Vary the focal length of the camera (i.e. zoom lens) Objects at different distances will become clear at different focal lengths Can use comparisons between pairs of images to get relative distances

17Ellen L. Walker Choice Depends on Object Solid, matte object (or matte separated from specular) Shape from shading, photometric stereo Highly reflective object Shape from specular reflection Regular textured object Shape from texture, stereo, focus Irregular textured object Stereo, focus