1. Computer Vision : CISC 4/689
Assignment
Program-2
Data
Due:
4/20/06 (Thu.) 10pm
Computer Vision : CISC 4/689

2. Computer Vision : CISC 4/689
CREDITS
Rasmussen, UBC (Jim Little), Seitz (U. of Wash.), Camps (Penn. State), UC, UMD (Jacobs), UNC, CUNY
Computer Vision : CISC 4/689

3. Computer Vision : CISC 4/689
Multi-View Geometry
3D World Points
Relates
Camera Orientations
Camera Centers
Computer Vision : CISC 4/689

4. Computer Vision : CISC 4/689
Multi-View Geometry
3D World Points
Relates
Camera Centers
Camera Intrinsic Parameters
Image Points
Camera Orientations
Computer Vision : CISC 4/689

5. Computer Vision : CISC 4/689
Stereo
scene point
image plane
optical center
Computer Vision : CISC 4/689

6. Computer Vision : CISC 4/689
Stereo
Basic Principle: Triangulation
Gives reconstruction as intersection of two rays
Requires
calibration
point correspondence
Computer Vision : CISC 4/689

7. Computer Vision : CISC 4/689
Stereo Constraints p’ p ?
Given p in left image, where can the corresponding point p’ in right image be?
Computer Vision : CISC 4/689

8. Computer Vision : CISC 4/689
Stereo Constraints M p’
Image plane
Epipolar Line Y2 X2 Z2 O2 Y1 p O1 Z1 X1
Epipole
Focal plane
Computer Vision : CISC 4/689

9. Computer Vision : CISC 4/689
Stereo
The geometric information that relates two different viewpoints of the same scene is entirely contained in a mathematical construct known as fundamental matrix.
The geometry of two different images of the same scene is called the epipolar geometry.
Computer Vision : CISC 4/689

10. Stereo/Two-View Geometry
The relationship of two views of a scene taken from different camera positions to one another
Interpretations
“Stereo vision” generally means two synchronized cameras or eyes capturing images
Could also be two sequential views from the same camera in motion
Assuming a static scene
Computer Vision : CISC 4/689

11. Computer Vision : CISC 4/689
3D from two-views
There are two ways of extracting 3D from a pair of images.
Classical method, called Calibrated route, we need to calibrate both cameras (or viewpoints) w.r.t some world coordinate system. i.e, calculate the so-called epipolar geometry by extracting the essential matrix of the system.
Second method, called uncalibrated route, a quantity known as fundamental matrix is calculated from image correspondences, and this is then used to determine the 3D.
Either way, principle of binocular vision is triangulation. Given a single image, the 3D location of any visible object point must lie on the straight line that passes through COP and image point (see fig.). Intersection of two such lines from two views is triangulation.
Computer Vision : CISC 4/689

12. Mapping Points between Images
What is the relationship between the images x, x’ of the scene point X in two views?
Intuitively, it depends on:
The rigid transformation between cameras (derivable from the camera matrices P, P’)
The scene structure (i.e., the depth of X)
Parallax: Closer points appear to move more
Computer Vision : CISC 4/689

13. Example: Two-View Geometry
courtesy of F. Dellaert
Is there a transformation relating the points xi to x’i ?
Computer Vision : CISC 4/689

14. Computer Vision : CISC 4/689
Epipolar Geometry
Baseline: Line joining camera centers C, C’
Epipolar plane ¦: Defined by baseline and scene point X
Computer Vision : CISC 4/689
baseline
from Hartley
& Zisserman

15. Computer Vision : CISC 4/689
Epipolar Lines
Epipolar lines l, l’: Intersection of epipolar plane ¦ with image planes
Epipoles e, e’: Where baseline intersects image planes
Equivalently, the image in one view of the other camera center. C C’
Computer Vision : CISC 4/689
from Hartley
& Zisserman

16. Computer Vision : CISC 4/689
Epipolar Pencil
As position of X varies, epipolar planes “rotate” about the baseline (like a book with pages)
This set of planes is called the epipolar pencil
Epipolar lines “radiate” from epipole—this is the pencil of epipolar lines
Computer Vision : CISC 4/689
from Hartley
& Zisserman

17. Computer Vision : CISC 4/689
Epipolar Constraint
Camera center C and image point define ray in 3-D space that projects to epipolar line l’ in other view (since it’s on the epipolar plane)
3-D point X on this ray, so image of X in other view x’ must be on l’
In other words, the epipolar geometry defines a mapping x ! l’, of points in one image to lines in the other x’ C C’
Computer Vision : CISC 4/689
from Hartley
& Zisserman

18. Example: Epipolar Lines for Converging Cameras
Left view
Right view
from Hartley
& Zisserman
Intersection of epipolar lines = Epipole !
Indicates direction of other camera
Computer Vision : CISC 4/689

19. Special Case: Translation Parallel to Image Plane
Note that epipolar lines are parallel and corresponding points lie on correspond-
ing epipolar lines (the latter is true for all kinds of camera motions)
Computer Vision : CISC 4/689

20. From Geometry to AlgebraP p p’
Computer Vision : CISC 4/689

21. From Geometry to AlgebraP p p’
Computer Vision : CISC 4/689

22. Computer Vision : CISC 4/689
Linear Constraint: Should be able to express as matrix multiplication.
Computer Vision : CISC 4/689

23. Review: Matrix Form of Cross Product
Computer Vision : CISC 4/689

24. Review: Matrix Form of Cross Product
Computer Vision : CISC 4/689

25. Computer Vision : CISC 4/689
Matrix Form
Computer Vision : CISC 4/689

26. Computer Vision : CISC 4/689
The Essential Matrix
Computer Vision : CISC 4/689

27. The Fundamental Matrix F
Mapping of point in one image to epipolar line in other image x ! l’ is expressed algebraically by the fundamental matrix F
Write this as l’ = F x
The point-on-line relationship is given by l’ ¢ Fx = 0, which can also be written as l’T Fx = (Fx)T l’ = 0
F is 3 x 3, rank 2 (not invertible, in contrast to homographies)
7 DOF (homogeneity and rank constraint take away 2 DOF)
line
point
Computer Vision : CISC 4/689

28. The Fundamental Matrix F
Since x’ is on l’, by the point-on-line definition we know that x’T l’ = 0
Combined with l’ = F x, we can thus relate corres-ponding points in the camera pair (P, P’) to each other with the following:
x’T F x = 0
The fundamental matrix of (P’, P) is the transpose FT x’
Computer Vision : CISC 4/689
from Hartley
& Zisserman

29. Computer Vision : CISC 4/689
Fundamental Matrix
(u’ is same as x in the prev. slide,
u’ is same as x)
Fundamental Matrix is singular with rank 2
In principal F has 7 parameters up to scale and can be estimated from 7 point correspondences
Direct Simpler Method requires 8 correspondences
Computer Vision : CISC 4/689

30. Estimating Fundamental Matrix
The 8-point algorithm
Each point correspondence can be expressed as a linear equation
Computer Vision : CISC 4/689

31. Computer Vision : CISC 4/689
The 8-point Algorithm
Computer Vision : CISC 4/689

32. Computing F: The Eight-point Algorithm
Input: n point correspondences ( n >= 8)
Construct homogeneous system Ax= 0 from
x = (f11,f12, ,f13, f21,f22,f23 f31,f32, f33) : entries in F
Each correspondence give one equation
A is a nx9 matrix
Obtain estimate F^ by SVD of A
x (up to a scale) is column of V corresponding to the least singular value
Enforce singularity constraint: since Rank (F) = 2
Compute SVD of F^
Set the smallest singular value to 0: D -> D’
Correct estimate of F :
Output: the estimate of the fundamental matrix, F’
Similarly we can compute E given intrinsic parameters
Why a homogeneous system?
Expand the F equation on blackboard
Computer Vision : CISC 4/689

33. Locating the Epipoles from F
el lies on all the epipolar lines of the left image
F is not identically zero
For every pr p l r P Ol Or el er Pl Pr
Epipolar Plane
Epipolar Lines
Epipoles
(1) Epipole on the left image
dot product of two vector pr , Fel = ql
F is not zero matrix
pr could be anything
ql must be zero vector
Fel = 0 => FtFel = 0
F = UDVt
Columns of V are the eigenvectors of FtF => The solution is the eigenvector corresponding to the null eigenvalue 0.
(2) Epipole on the right image
Do the similar thing to er
Input: Fundamental Matrix F
Find the SVD of F
The epipole el is the column of V corresponding to the null singular value (as shown above)
The epipole er is the column of U corresponding to the null singular value
Output: Epipole el and er
Computer Vision : CISC 4/689

34. Computer Vision : CISC 4/689
Stereo Rectification p’ l r P Ol Or
X’r Pl Pr
Z’l
Y’l
Y’r T
X’l
Z’r
Stereo System with Parallel Optical Axes
Epipoles are at infinity
Horizontal epipolar lines
Rectification
Given a stereo pair, the intrinsic and extrinsic parameters, find the image transformation to achieve a stereo system of horizontal epipolar lines
A simple algorithm: Assuming calibrated stereo cameras
Computer Vision : CISC 4/689

35. Computer Vision : CISC 4/689
Stereo Rectification p l r P Ol Or Xl Xr Pl Pr Zl Yl Zr Yr
R, T T
X’l
Xl’ = T, Yl’ = Xl’xZl, Z’l = Xl’xYl’
Algorithm
Rotate both left and right camera so that they share the same X axis : Or-Ol = T
Define a rotation matrix Rrect for the left camera
Rotation Matrix for the right camera is RrectRT
Rotation can be implemented by image transformation
Pr = R (Pl – T)
Pl’ = Rrect Pl
Pr’ = Rrect RT Pr
First rotate the right camera so that it’s three axes parallel to the left camera; then
Rotate both camera so that they have the same x axis, and Z axes are parallel
Computer Vision : CISC 4/689

36. Computer Vision : CISC 4/689
Stereo Rectification p l r P Ol Or Xl Xr Pl Pr Zl Yl Zr Yr
R, T T
X’l
Xl’ = T, Yl’ = Xl’xZl, Z’l = Xl’xYl’
Algorithm
Rotate both left and right camera so that they share the same X axis : Or-Ol = T
Define a rotation matrix Rrect for the left camera
Rotation Matrix for the right camera is RrectRT
Rotation can be implemented by image transformation
Pr = R (Pl – T)
Pl’ = Rrect Pl
Pr’ = Rrect RT Pr
First rotate the right camera so that it’s three axes parallel to the left camera; then
Rotate both camera so that they have the same x axis, and Z axes are parallel
Computer Vision : CISC 4/689

37. Computer Vision : CISC 4/689
Stereo Rectification Zr p’ l r P Ol Or
X’r Pl Pr
Z’l
Y’l
Y’r
R, T T
X’l
T’ = (B, 0, 0), P’r = P’l – T’
Algorithm
Rotate both left and right camera so that they share the same X axis : Or-Ol = T
Define a rotation matrix Rrect for the left camera
Rotation Matrix for the right camera is RrectRT
Rotation can be implemented by image transformation
Pr = R (Pl – T)
Pl’ = Rrect Pl
Pr’ = Rrect RT Pr
First rotate the right camera so that it’s three axes parallel to the left camera; then
Rotate both camera so that they have the same x axis, and Z axes are parallel
Computer Vision : CISC 4/689

38. Special Case: Translation along Optical Axis
Epipoles coincide at focus of expansion
Not the same (in general) as vanishing point of scene lines
Computer Vision : CISC 4/689
from Hartley & Zisserman

39. Finding Correspondences
Epipolar geometry limits where feature in one image can be in the other image
Only have to search along a line
Computer Vision : CISC 4/689

40. Computer Vision : CISC 4/689
Simplest Case
Image planes of cameras are parallel.
Focal points are at same height.
Focal lengths same.
Then, epipolar lines are horizontal scan lines.
Computer Vision : CISC 4/689

41. We can always achieve this geometry with image rectification
Image Reprojection
reproject image planes onto common plane parallel to line between optical centers
Notice, only focal point of camera really matters
Computer Vision : CISC 4/689
(Seitz)

42. Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
Computer Vision : CISC 4/689

43. Teesta suspension bridge-Darjeeling, India
Computer Vision : CISC 4/689

44. Mark Twain at Pool Table", no date, UCR Museum of Photography
Computer Vision : CISC 4/689

45. Computer Vision : CISC 4/689
Woman getting eye exam during immigration procedure at Ellis Island, c , UCR Museum of Phography
Computer Vision : CISC 4/689

46. Computer Vision : CISC 4/689
Stereo matching
attempt to match every pixel
use additional constraints
Computer Vision : CISC 4/689

47. Computer Vision : CISC 4/689
A Simple Stereo System
LEFT CAMERA
RIGHT CAMERA
baseline
Elevation Zw
disparity
Depth Z
Right image:
target
Left image:
reference
Zw=0
Computer Vision : CISC 4/689

48. Computer Vision : CISC 4/689
Let’s discuss reconstruction with this geometry before correspondence, because it’s much easier. P
{T-xl-xr}/{z-f}={T}/{Z}
{T-xl-(d+xl)}/{z-f}={T}/{Z}
{T- d}/{z-f}={T}/{Z} Z
Disparity: xl xr f pl pr Ol Or T
Then given Z, we can compute X and Y.
T is the stereo baseline
d measures the difference in retinal position between corresponding points
(Camps)
Computer Vision : CISC 4/689

49. Correspondence: What should we match?
Objects?
Edges?
Pixels?
Collections of pixels?
Computer Vision : CISC 4/689

50. Computer Vision : CISC 4/689
Extracting Structure
The key aspect of epipolar geometry is its linear constraint on where a point in one image can be in the other
By correlation-matching pixels (or features) along epipolar lines and measuring the disparity between them, we can construct a depth map (scene point depth is inversely proportional to disparity)
courtesy of P. Debevec
View 1
View 2
Computed depth map
Computer Vision : CISC 4/689

51. Correspondence: Photometric constraint
Same world point has same intensity in both images.
Lambertian fronto-parallel
Issues:
Noise
Specularity
Foreshortening
Computer Vision : CISC 4/689

52. Using these constraints we can use matching for stereo
Improvement: match windows
For each pixel in the left image
For each epipolar line
compare with every pixel on same epipolar line in right image
pick pixel with minimum match cost
This will never work, so:
(Seitz)
Computer Vision : CISC 4/689

53. Computer Vision : CISC 4/689= ? f g
Comparing Windows:
Most
popular
For each window, match to closest window on epipolar line in other image.
(Camps)
Computer Vision : CISC 4/689

54. Computer Vision : CISC 4/689
Aggregation
Use more than one pixel
Assume neighbors have similar disparities*
Use correlation window containing pixel
Allows to use SSD, ZNCC, etc.
Computer Vision : CISC 4/689

55. Comparing image regions
Compare intensities pixel-by-pixel
I(x,y)
I´(x,y)
Dissimilarity measures
Sum of Square Differences
Computer Vision : CISC 4/689

56. Comparing image regions
Compare intensities pixel-by-pixel
I(x,y)
I´(x,y)
Similarity measures
Zero-mean Normalized Cross Correlation
Computer Vision : CISC 4/689

57. Aggregation window sizes
Small windows
disparities similar
more ambiguities
accurate when correct
Large windows
larger disp. variation
more discriminant
often more robust
use shiftable windows to deal with discontinuities
Computer Vision : CISC 4/689
(Illustration from Pascal Fua)

58. Computer Vision : CISC 4/689
Window size
W = 3
W = 20
Effect of window size
Better results with adaptive window
T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.
D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2): , July 1998
smaller window: more detail, more noise
bigger window: less noise, more detail
(Seitz)
Computer Vision : CISC 4/689

59. Correspondence Using Window-based matching
scanline
Left
Right
SSD error
disparity
Computer Vision : CISC 4/689
Left
Right

60. Sum of Squared (Pixel) Differences
Left
Right
Computer Vision : CISC 4/689

61. Computer Vision : CISC 4/689
Image Normalization
Even when the cameras are identical models, there can be differences in gain and sensitivity.
The cameras do not see exactly the same surfaces, so their overall light levels can differ.
For these reasons and more, it is a good idea to normalize the pixels in each window:
Computer Vision : CISC 4/689

62. Computer Vision : CISC 4/689
Stereo results
Data from University of Tsukuba
Scene
Ground truth
(Seitz)
Computer Vision : CISC 4/689

63. Results with window correlation
Window-based matching
(best window size)
Ground truth
(Seitz)
Computer Vision : CISC 4/689

64. Results with better method
State of the art method
Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
Computer Vision : CISC 4/689
(Seitz)

65. Computer Vision : CISC 4/689
NOW TO ASSIGNMENT!
Computer Vision : CISC 4/689