1. Multiple View Geometry
Marc Pollefeys
COMP 256

2. Last class Gaussian pyramid Laplacian pyramid Gabor Fourier filters
transform
Texture
synthesis

3. Not last class…

4. Shape-from-texture

5. Tentative class schedule
Aug 26/28 -
Introduction
Sep 2/4
Cameras
Radiometry
Sep 9/11
Sources & Shadows
Color
Sep 16/18
Linear filters & edges
(Isabel hurricane)
Sep 23/25
Pyramids & Texture
Multi-View Geometry
Sep30/Oct2
Stereo
Project proposals
Oct 7/9
Optical flow
Oct 14/16
Tracking
Oct 21/23
Silhouettes/carving
Structure from motion
Oct 28/30
Camera calibration
Nov 4/6
Project update
Segmentation
Nov 11/13
Fitting
Probabilistic segm.&fit.
Nov 18/20
Matching templates
Matching relations
Nov 25/27
Range data
(Thanksgiving)
Dec 2/4
Final project

6. THE GEOMETRY OF MULTIPLE VIEWS
Epipolar Geometry
The Essential Matrix
The Fundamental Matrix
The Trifocal Tensor
The Quadrifocal Tensor
Reading: Chapter 10.

7. Epipolar Geometry
Epipolar Plane
Baseline
Epipoles
Epipolar Lines

8. Potential matches for p have to lie on the corresponding
Epipolar Constraint
Potential matches for p have to lie on the corresponding
epipolar line l’.
Potential matches for p’ have to lie on the corresponding
epipolar line l.

9. Epipolar Constraint: Calibrated Case
Essential Matrix
(Longuet-Higgins, 1981)

10. E p’ is the epipolar line associated with p’.
Properties of the Essential Matrix T
E p’ is the epipolar line associated with p’.
ETp is the epipolar line associated with p.
E e’=0 and ETe=0.
E is singular.
E has two equal non-zero singular values
(Huang and Faugeras, 1989). T

11. Epipolar Constraint: Small Motions
To First-Order:
Pure translation:
Focus of Expansion

12. Epipolar Constraint: Uncalibrated Case
Fundamental Matrix
(Faugeras and Luong, 1992)

13. Properties of the Fundamental Matrix
F p’ is the epipolar line associated with p’.
FT p is the epipolar line associated with p.
F e’=0 and FT e=0.
F is singular. T T

14. The Eight-Point Algorithm (Longuet-Higgins, 1981)
|F | =1.
Minimize:
under the constraint 2

15. Non-Linear Least-Squares Approach
(Luong et al., 1993)
Minimize
with respect to the coefficients of F , using an
appropriate rank-2 parameterization.

16. Problem with eight-point algorithm
linear least-squares:
unit norm vector F yielding smallest residual
What happens when there is noise?

17. The Normalized Eight-Point Algorithm (Hartley, 1995)
Center the image data at the origin, and scale it so the
mean squared distance between the origin and the data
points is 2 pixels: q = T p , q’ = T’ p’.
Use the eight-point algorithm to compute F from the
points q and q’ .
Enforce the rank-2 constraint.
Output T F T’. i i i i i i T

18. Epipolar geometry example

19. courtesy of Andrew Zisserman
Example: converging cameras
courtesy of Andrew Zisserman

20. Example: motion parallel with image plane
(simple for stereo  rectification)
courtesy of Andrew Zisserman

21. courtesy of Andrew Zisserman
Example: forward motion e’ e
courtesy of Andrew Zisserman

22. courtesy of Andrew Zisserman
Fundamental matrix for pure translation
auto-epipolar
courtesy of Andrew Zisserman

23. courtesy of Andrew Zisserman
Fundamental matrix for pure translation
courtesy of Andrew Zisserman

24. Trinocular Epipolar Constraints
These constraints are not independent!

25. Trinocular Epipolar Constraints: Transfer
Given p and p , p can be computed
as the solution of linear equations. 1 2 3

26. Trinocular Epipolar Constraints: Transfer
problem for epipolar transfer in trifocal plane!
There must be more to trifocal geometry…
image from Hartley and Zisserman

27. Trifocal Constraints

28. Trifocal Constraints Calibrated Case All 3x3 minors must be zero!
Trifocal Tensor

29. Trifocal Constraints
Uncalibrated Case
Trifocal Tensor

30. Trifocal Constraints: 3 Points
Pick any two lines l and l through p and p . 2 3 2 3
Do it again.
T( p , p , p )=0 1 2 3

31. For any matching epipolar lines, l G l = 0.
Properties of the Trifocal Tensor
For any matching epipolar lines, l G l = 0.
The matrices G are singular.
They satisfy 8 independent constraints in the
uncalibrated case (Faugeras and Mourrain, 1995). T i 2 1 3 i 1
Estimating the Trifocal Tensor
Ignore the non-linear constraints and use linear least-squares
Impose the constraints a posteriori.

32. For any matching epipolar lines, l G l = 0.2 1 3
The backprojections of the two lines do not define a line!

33. courtesy of Andrew Zisserman
Trifocal
Tensor
Example
108 putative matches
18 outliers
(26 samples)
88 inliers
95 final inliers
(0.43)
(0.23)
(0.19)
courtesy of Andrew Zisserman

34. Trifocal Tensor Example
additional line matches
images courtesy of Andrew Zisserman

35. Transfer: trifocal transfer
(using tensor notation)
doesn’t work if l’=epipolar line
image courtesy of Hartley and Zisserman

36. Image warping using T(1,2,N)
(Avidan and Shashua `97)

37. Multiple Views (Faugeras and Mourrain, 1995)

38. Two Views
Epipolar Constraint

39. Three Views
Trifocal Constraint

40. Four Views
Quadrifocal Constraint
(Triggs, 1995)

41. Geometrically, the four rays must intersect in P..

42. Quadrifocal Tensor
and Lines

43. Quadrifocal tensor determinant is multilinear
thus linear in coefficients of lines !
There must exist a tensor with 81 coefficients containing all possible combination of x,y,w coefficients for all 4 images: the quadrifocal tensor

44. Scale-Restraint Condition from Photogrammetry

45. Next class: Stereo (x´,y´)=(x+D(x,y),y) F&P Chapter 11 image I´(x´,y´)
Disparity map D(x,y)
image I´(x´,y´)
(x´,y´)=(x+D(x,y),y)
F&P
Chapter 11