3D Imaging Midterm Review

Pinhole cameras Abstract camera model - box with a small hole in it Pinhole cameras work in practice The point to make here is that each point on the image plane sees light from only one direction, the one that passes through the pinhole.

The equation of projection Cartesian coordinates: We have, by similar triangles, that (x, y, z) -> (f x/z, f y/z, -f) Ignore the third coordinate, and get Prove at home

The camera matrix Homogenous coordinates for 3D point are (X,Y,Z,T) Homogenous coordinates for point in image are (U,V,W)

Properties of “thin” lens (i.e., ideal lens) Light rays passing through the center are not deviated. Light rays passing through a point far away from the center are deviated more. focal point f 6

Thin lens equation (cont’d) Combining the equations(do at home): v u f And the set of all such points forms a plane parallel to the image (plane of focus). 1 u v f + = image 7

Thin lens equation (cont’d) The thin lens equation implies that only points at distance u from the lens are “in focus” (i.e., focal point lies on image plane). Other points project to a “blur circle” or “circle of confusion” in the image (i.e., blurring occurs). “circle of confusion” 8

Depth of field f / 5.6 f / 32 Changing the aperture size or focal length affects depth of field

Basic models of reflection Specular: light bounces off at the incident angle E.g., mirror Diffuse: light scatters in all directions E.g., brick, cloth, rough wood specular reflection incoming light Θ Θ diffuse reflection incoming light

Bidirectional Reflectance Distribution Function (BDRF) Model of local reflection that tells how bright a surface appears when viewed from one direction when light falls on it from another surface normal

The Retina

Two types of light-sensitive receptors Cones cone-shaped less sensitive operate in high light color vision Rods rod-shaped highly sensitive operate at night gray-scale vision © Stephen E. Palmer, 2002

Color Vision

The raster image (pixel matrix) 0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.99 0.95 0.89 0.82 0.56 0.31 0.75 0.81 0.91 0.72 0.51 0.55 0.42 0.57 0.41 0.49 0.96 0.88 0.46 0.87 0.90 0.71 0.80 0.79 0.60 0.58 0.50 0.61 0.45 0.33 0.86 0.84 0.74 0.39 0.73 0.67 0.54 0.48 0.69 0.66 0.43 0.77 0.78

Color Images: Multi-chip Bulky and expensive wavelength dependent

Color Images: Bayer Grid Estimate RGB at ‘G’ cells from neighboring values Slide by Steve Seitz

HSV Color Space

Moravec corner detector Change of intensity for the shift [u,v]: Intensity Window function Shifted intensity Four shifts: (u,v) = (1,0), (1,1), (0,1), (-1, 1) Look for local maxima in min{E}

Harris Corner Detector

Image filtering 1 90 90 ones, divide by 9

Practice with linear filters - 2 1 Original Sharpening filter Accentuates differences with local average Source: D. Lowe

Other filters -1 1 -2 2 Sobel Vertical Edge (absolute value)

Important filter: Gaussian Spatially-weighted average 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 5 x 5,  = 1

Smoothing with Gaussian filter

Smoothing with box filter

A sum of sines Our building block: Add enough of them to get any signal f(x) you want!

Fourier analysis in images Intensity Image Fourier Image

Gaussian

Box Filter

Aliasing problem 1D example (sinewave): Source: S. Marschner

Aliasing in video

Aliasing in graphics

Subsampling without pre-filtering 1/2 1/4 (2x zoom) 1/8 (4x zoom)

Subsampling with Gaussian pre-filtering

Template matching Goal: find in image Main challenge: What is a good similarity or distance measure between two patches? Correlation Zero-mean correlation Sum Square Difference Normalized Cross Correlation

Normalized Cross Correlation mean template mean image patch Invariant to mean and scale of intensity

Matching with filters (Normalized Cross Correlation) True detections Input Thresholded Image Normalized X-Correlation

Reducing salt-and-pepper noise by Gaussian smoothing 3x3 5x5 7x7 What’s wrong with these results?

Alternative idea: Median filtering A median filter operates over a window by selecting the median intensity in the window Better at salt’n’pepper noise Not convolution: try a region with 1’s and a 2, and then 1’s and a 3 Is median filtering linear?

Salt-and-pepper noise Median filter Salt-and-pepper noise Median filtered

Effects of noise Where is the edge? Consider a single row or column of the image Plotting intensity as a function of position gives a signal Where is the edge? How to fix?

Solution: smooth first g f * g To find edges, look for peaks in Source: S. Seitz

Derivative theorem of convolution Differentiation is convolution, and convolution is associative: : f

Derivative of Gaussian filter

Final Canny Edges

Estimating Camera Parameters Alper Yilmaz, CAP5415, Fall 2004

Ames Room

Julesz: had huge impact because it showed that recognition not needed for stereo.

Epipolar Constraint

Basic Stereo Derivations Disparity:

We can always achieve this geometry with image rectification Image Reprojection reproject image planes onto common plane parallel to line between optical centers (Seitz)

Using these constraints we can use matching for stereo Improvement: match windows For each pixel in the left image For each epipolar line compare with every pixel on same epipolar line in right image pick pixel with minimum match cost This will never work, so:

Stereo matching as energy minimization D W1(i ) W2(i+D(i )) D(i ) Energy functions of this form can be minimized using graph cuts Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001

Active stereo with structured light L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002

Random Dot

Time to Collision v Can be directly measured from image L L f t t=0 Do l(t) f t t=0 Do And time to collision: D(t) Can be found, without knowing L or Do or v !!

2D Motion Field

2D Optical Flow Apparent motion of image brightness pattern

2D Motion Field and 2D Optical Flow Motion field: projection of 3D motion vectors on image plane Optical flow field: apparent motion of brightness patterns We equate motion field with optical flow field

Brightness Constancy Equation Taking derivative wrt time:

Normal Motion/Aperture Problem

Barber Pole Illusion A: u and v are unknown, 1 equation

Full 3D Rotation Any rotation can be expressed as combination of three rotations about three axes. Rows (and columns) of R are orthonormal vectors. R has determinant 1 (not -1).

Velocity Model in 2D Perspective projection