1. Image Primitives and Correspondence
Stefano Soatto added with slides from Univ. of Maryland and
R.Bajcsy, UCB
Computer Science Department
University of California at Los Angeles

2. Image Primitives and Correspondence
Given an image point in left image, what is the (corresponding) point in the right
image, which is the projection of the same 3-D point
Siggraph 04

3. Image primitives and Features
The desirable properties of features are:
Invariance with respect to Grays scale/color
With respect to location (translation and rotation)
With respect to scale
Robustness
Easy to compute
Local features vs. global features
Siggraph 04

4. Feature analysis
Points sensitive to illumination variation but fast to compute
Neighborhood features : gradient based (edge detectors) measuring contrast ,robust to illumination variation except for highlights
Fast computation ,it can be done in parallel.
The complimentary feature to gradient is region based. The advantage of this feature is it can encompass larger regions that are homogeneous and save processing time.
Siggraph 04

5. Profiles of image intensity edges
Siggraph 04

6. Image gradient The gradient of an image:
The gradient points in the direction of most rapid change in intensity
The gradient direction is given by:
how does this relate to the direction of the edge?
The edge strength is given by the gradient magnitude
Siggraph 04

7. How can we differentiate a digital image f[x,y]?
The discrete gradient
How can we differentiate a digital image f[x,y]?
Option 1: reconstruct a continuous image, then take gradient
Option 2: take discrete derivative (finite difference)
How would you implement this as a cross-correlation?
Siggraph 04

8. Consider a single row or column of the image
Effects of noise
Consider a single row or column of the image
Plotting intensity as a function of position gives a signal
Where is the edge?
Siggraph 04

9. Solution: smooth first
Siggraph 04
Where is the edge?
Look for peaks in

10. 2D edge detection filters
Laplacian of Gaussian
Gaussian
derivative of Gaussian
is the Laplacian operator:
Siggraph 04

11. Effect of  (Gaussian kernel size)
original
Canny with
Canny with
The choice of depends on desired behavior
large detects large scale edges
small detects fine features
Siggraph 04

12. Eliminates noise edges. Makes edges smoother. Removes fine detail.
Scale
Smoothing
Eliminates noise edges.
Makes edges smoother.
Removes fine detail.
Figures show gradient magnitude of zebra at two different scales
Siggraph 04
(Forsyth & Ponce)

13. Corners contain more edges than lines.
Corner detection
Corners contain more edges than lines.
A point on a line is hard to match.
Siggraph 04

14. Corners contain more edges than lines.
A corner is easier
Siggraph 04

15. Edge Detectors Tend to Fail at Corners
Siggraph 04

16. Right at corner, gradient is ill defined.
Finding Corners
Intuition:
Right at corner, gradient is ill defined.
Near corner, gradient has two different values.
Siggraph 04

17. Formula for Finding Corners
We look at matrix:
Gradient with respect to x, times gradient with respect to y
Sum over a small region, the hypothetical corner
WHY THIS?
Matrix is symmetric
Siggraph 04

18. First, consider case where:
What is region like if:
l1 = 0?
l2 = 0?
l1 = 0 and l2 = 0?
l1 > 0 and l2 > 0?
Siggraph 04

19. General Case:
From Linear Algebra, it follows that because C is symmetric:
With R a rotation matrix.
So every case is like one on last slide.
Siggraph 04

20. Compute magnitude of the gradient everywhere.
So, to detect corners
Filter image.
Compute magnitude of the gradient everywhere.
We construct C in a window.
Use Linear Algebra to find l1 and l2.
If they are both big, we have a corner.
Siggraph 04

21. Matching - Correspondence
Lambertian assumption
Rigid body motion
Correspondence
Siggraph 04

22. Local Deformation Models
Translational model
Affine model
Transformation of the intensity values and occlusions
Siggraph 04

23. The MF assigns a velocity vector to each pixel in the image.
Motion Field (MF)
The MF assigns a velocity vector to each pixel in the image.
These velocities are INDUCED by the RELATIVE MOTION btw the camera and the 3D scene
The MF can be thought as the projection of the 3D velocities on the image plane.
Siggraph 04

24. Motion Field and Optical Flow Field
Motion field: projection of 3D motion vectors on image plane
Optical flow field: apparent motion of brightness patterns
We equate motion field with optical flow field
Siggraph 04

25. 2 Cases Where this Assumption Clearly is not Valid
(a) A smooth sphere is rotating under constant illumination. Thus the optical flow field is zero, but the motion field is not.
(b) A fixed sphere is illuminated by a moving source—the shading of the image changes. Thus the motion field is zero, but the optical flow field is not.
(a) (b)
Siggraph 04

26. Brightness Constancy Equation
Let P be a moving point in 3D:
At time t, P has coords (X(t),Y(t),Z(t))
Let p=(x(t),y(t)) be the coords. of its image at time t.
Let E(x(t),y(t),t) be the brightness at p at time t.
Brightness Constancy Assumption:
As P moves over time, E(x(t),y(t),t) remains constant.
Siggraph 04

27. Brightness Constraint Equation
short:
Siggraph 04

28. Brightness Constancy Equation
Taking derivative wrt time:
Siggraph 04

29. Brightness Constancy Equation
Let
(Frame spatial gradient)
(optical flow)
(derivative across frames)
and
Siggraph 04

30. Brightness Constancy Equation
Becomes: vy
r E
-Et/|r E| vx
The OF is CONSTRAINED to be on a line !
Siggraph 04

31. Interpretation
Values of (u, v) satisfying the constraint equation lie on a straight line in velocity space. A local measurement only provides this constraint line (aperture problem).
Siggraph 04

32. Aperture Problem
Normal flow
Siggraph 04

33. Recall the corner detector
The matrix for corner detection:
is singular (not invertible) when det(ATA) = 0
But det(ATA) = Õ li = 0 -> one or both e.v. are 0
Aperture Problem !
One e.v. = 0 -> no corner, just an edge
Two e.v. = 0 -> no corner, homogeneous region
Siggraph 04

34. Optical Flow
Integrate over image patch
Solve
Siggraph 04

35. Optical Flow, Feature Tracking
Conceptually:
rank(G) = 0 blank wall problem
rank(G) = 1 aperture problem
rank(G) = 2 enough texture – good feature candidates
In reality: choice of threshold is involved
Siggraph 04

36. Optical Flow Previous method - assumption locally constant flow
Alternative regularization techniques (locally smooth flow fields,
integration along contours)
Qualitative properties of the motion fields
Siggraph 04

37. Feature Tracking
Siggraph 04

38. 3D Reconstruction - Preview
Siggraph 04

39. Harris Corner Detector - Example
Siggraph 04

40. Wide Baseline Matching
Siggraph 04

41. Region based Similarity Metric
Sum of squared differences
Normalize cross-correlation
Sum of absolute differences
Siggraph 04

42. Edge Detection Canny edge detector Compute image derivatives
original image
gradient magnitude
Canny edge detector
Compute image derivatives
if gradient magnitude >  and the value is a local maximum along gradient
direction – pixel is an edge candidate
Siggraph 04

43. Line fitting Edge detection, non-maximum suppression  x y
Non-max suppressed gradient magnitude
Edge detection, non-maximum suppression
(traditionally Hough Transform – issues of resolution, threshold
selection and search for peaks in Hough space)
Connected components on edge pixels with similar orientation
- group pixels with common orientation
Siggraph 04

44. Line Fitting second moment matrix associated with each
connected component
Line fitting Lines determined from eigenvalues and eigenvectors of A
Candidate line segments - associated line quality
Siggraph 04

45. From now on just geometry
Take home messages
Correspondence is easy/difficult/impossible depending on the imaging constraints
Correspondence and reconstruction are tightly coupled problems, can be solved simultaneously [Jin et al., CVPR 2004]
For most scenes simple descriptors suffice to establish a few (50-500) corresponding points/lines
From now on just geometry
Siggraph 04