Image Primitives and Correspondence Stefano Soatto added with slides from Univ. of Maryland and R.Bajcsy, UCB Computer Science Department University of California at Los Angeles

Image Primitives and Correspondence Given an image point in left image, what is the (corresponding) point in the right image, which is the projection of the same 3-D point Siggraph 04

Image primitives and Features The desirable properties of features are: Invariance with respect to Grays scale/color With respect to location (translation and rotation) With respect to scale Robustness Easy to compute Local features vs. global features Siggraph 04

Feature analysis Points sensitive to illumination variation but fast to compute Neighborhood features : gradient based (edge detectors) measuring contrast ,robust to illumination variation except for highlights Fast computation ,it can be done in parallel. The complimentary feature to gradient is region based. The advantage of this feature is it can encompass larger regions that are homogeneous and save processing time. Siggraph 04

Profiles of image intensity edges Siggraph 04

Image gradient The gradient of an image: The gradient points in the direction of most rapid change in intensity The gradient direction is given by: how does this relate to the direction of the edge? The edge strength is given by the gradient magnitude Siggraph 04

How can we differentiate a digital image f[x,y]? The discrete gradient How can we differentiate a digital image f[x,y]? Option 1: reconstruct a continuous image, then take gradient Option 2: take discrete derivative (finite difference) How would you implement this as a cross-correlation? Siggraph 04

Consider a single row or column of the image Effects of noise Consider a single row or column of the image Plotting intensity as a function of position gives a signal Where is the edge? Siggraph 04

Solution: smooth first Siggraph 04 Where is the edge? Look for peaks in

2D edge detection filters Laplacian of Gaussian Gaussian derivative of Gaussian is the Laplacian operator: Siggraph 04

Effect of  (Gaussian kernel size) original Canny with Canny with The choice of depends on desired behavior large detects large scale edges small detects fine features Siggraph 04

Eliminates noise edges. Makes edges smoother. Removes fine detail. Scale Smoothing Eliminates noise edges. Makes edges smoother. Removes fine detail. Figures show gradient magnitude of zebra at two different scales Siggraph 04 (Forsyth & Ponce)

Corners contain more edges than lines. Corner detection Corners contain more edges than lines. A point on a line is hard to match. Siggraph 04

Corners contain more edges than lines. A corner is easier Siggraph 04

Edge Detectors Tend to Fail at Corners Siggraph 04

Right at corner, gradient is ill defined. Finding Corners Intuition: Right at corner, gradient is ill defined. Near corner, gradient has two different values. Siggraph 04

Formula for Finding Corners We look at matrix: Gradient with respect to x, times gradient with respect to y Sum over a small region, the hypothetical corner WHY THIS? Matrix is symmetric Siggraph 04

First, consider case where: What is region like if: l1 = 0? l2 = 0? l1 = 0 and l2 = 0? l1 > 0 and l2 > 0? Siggraph 04

General Case: From Linear Algebra, it follows that because C is symmetric: With R a rotation matrix. So every case is like one on last slide. Siggraph 04

Compute magnitude of the gradient everywhere. So, to detect corners Filter image. Compute magnitude of the gradient everywhere. We construct C in a window. Use Linear Algebra to find l1 and l2. If they are both big, we have a corner. Siggraph 04

Matching - Correspondence Lambertian assumption Rigid body motion Correspondence Siggraph 04

Local Deformation Models Translational model Affine model Transformation of the intensity values and occlusions Siggraph 04

The MF assigns a velocity vector to each pixel in the image. Motion Field (MF) The MF assigns a velocity vector to each pixel in the image. These velocities are INDUCED by the RELATIVE MOTION btw the camera and the 3D scene The MF can be thought as the projection of the 3D velocities on the image plane. Siggraph 04

Motion Field and Optical Flow Field Motion field: projection of 3D motion vectors on image plane Optical flow field: apparent motion of brightness patterns We equate motion field with optical flow field Siggraph 04

2 Cases Where this Assumption Clearly is not Valid (a) A smooth sphere is rotating under constant illumination. Thus the optical flow field is zero, but the motion field is not. (b) A fixed sphere is illuminated by a moving source—the shading of the image changes. Thus the motion field is zero, but the optical flow field is not. (a) (b) Siggraph 04

Brightness Constancy Equation Let P be a moving point in 3D: At time t, P has coords (X(t),Y(t),Z(t)) Let p=(x(t),y(t)) be the coords. of its image at time t. Let E(x(t),y(t),t) be the brightness at p at time t. Brightness Constancy Assumption: As P moves over time, E(x(t),y(t),t) remains constant. Siggraph 04

Brightness Constraint Equation short: Siggraph 04

Brightness Constancy Equation Taking derivative wrt time: Siggraph 04

Brightness Constancy Equation Let (Frame spatial gradient) (optical flow) (derivative across frames) and Siggraph 04

Brightness Constancy Equation Becomes: vy r E -Et/|r E| vx The OF is CONSTRAINED to be on a line ! Siggraph 04

Interpretation Values of (u, v) satisfying the constraint equation lie on a straight line in velocity space. A local measurement only provides this constraint line (aperture problem). Siggraph 04

Aperture Problem Normal flow Siggraph 04

Recall the corner detector The matrix for corner detection: is singular (not invertible) when det(ATA) = 0 But det(ATA) = Õ li = 0 -> one or both e.v. are 0 Aperture Problem ! One e.v. = 0 -> no corner, just an edge Two e.v. = 0 -> no corner, homogeneous region Siggraph 04

Optical Flow Integrate over image patch Solve Siggraph 04

Optical Flow, Feature Tracking Conceptually: rank(G) = 0 blank wall problem rank(G) = 1 aperture problem rank(G) = 2 enough texture – good feature candidates In reality: choice of threshold is involved Siggraph 04

Optical Flow Previous method - assumption locally constant flow Alternative regularization techniques (locally smooth flow fields, integration along contours) Qualitative properties of the motion fields Siggraph 04

Feature Tracking Siggraph 04

3D Reconstruction - Preview Siggraph 04

Harris Corner Detector - Example Siggraph 04

Wide Baseline Matching Siggraph 04

Region based Similarity Metric Sum of squared differences Normalize cross-correlation Sum of absolute differences Siggraph 04

Edge Detection Canny edge detector Compute image derivatives original image gradient magnitude Canny edge detector Compute image derivatives if gradient magnitude >  and the value is a local maximum along gradient direction – pixel is an edge candidate Siggraph 04

Line fitting Edge detection, non-maximum suppression   x y Non-max suppressed gradient magnitude Edge detection, non-maximum suppression (traditionally Hough Transform – issues of resolution, threshold selection and search for peaks in Hough space) Connected components on edge pixels with similar orientation - group pixels with common orientation Siggraph 04

Line Fitting second moment matrix associated with each connected component Line fitting Lines determined from eigenvalues and eigenvectors of A Candidate line segments - associated line quality Siggraph 04

From now on just geometry Take home messages Correspondence is easy/difficult/impossible depending on the imaging constraints Correspondence and reconstruction are tightly coupled problems, can be solved simultaneously [Jin et al., CVPR 2004] For most scenes simple descriptors suffice to establish a few (50-500) corresponding points/lines From now on just geometry Siggraph 04