Multiple View Geometry Marc Pollefeys COMP 256

Last class Gaussian pyramid Laplacian pyramid Gabor Fourier filters transform Texture synthesis

Not last class…

Shape-from-texture

Tentative class schedule Aug 26/28 - Introduction Sep 2/4 Cameras Radiometry Sep 9/11 Sources & Shadows Color Sep 16/18 Linear filters & edges (Isabel hurricane) Sep 23/25 Pyramids & Texture Multi-View Geometry Sep30/Oct2 Stereo Project proposals Oct 7/9 Optical flow Oct 14/16 Tracking Oct 21/23 Silhouettes/carving Structure from motion Oct 28/30 Camera calibration Nov 4/6 Project update Segmentation Nov 11/13 Fitting Probabilistic segm.&fit. Nov 18/20 Matching templates Matching relations Nov 25/27 Range data (Thanksgiving) Dec 2/4 Final project

THE GEOMETRY OF MULTIPLE VIEWS Epipolar Geometry The Essential Matrix The Fundamental Matrix The Trifocal Tensor The Quadrifocal Tensor Reading: Chapter 10.

Epipolar Geometry Epipolar Plane Baseline Epipoles Epipolar Lines

Potential matches for p have to lie on the corresponding Epipolar Constraint Potential matches for p have to lie on the corresponding epipolar line l’. Potential matches for p’ have to lie on the corresponding epipolar line l.

Epipolar Constraint: Calibrated Case Essential Matrix (Longuet-Higgins, 1981)

E p’ is the epipolar line associated with p’. Properties of the Essential Matrix T E p’ is the epipolar line associated with p’. ETp is the epipolar line associated with p. E e’=0 and ETe=0. E is singular. E has two equal non-zero singular values (Huang and Faugeras, 1989). T

Epipolar Constraint: Small Motions To First-Order: Pure translation: Focus of Expansion

Epipolar Constraint: Uncalibrated Case Fundamental Matrix (Faugeras and Luong, 1992)

Properties of the Fundamental Matrix F p’ is the epipolar line associated with p’. FT p is the epipolar line associated with p. F e’=0 and FT e=0. F is singular. T T

The Eight-Point Algorithm (Longuet-Higgins, 1981) |F | =1. Minimize: under the constraint 2

Non-Linear Least-Squares Approach (Luong et al., 1993) Minimize with respect to the coefficients of F , using an appropriate rank-2 parameterization.

Problem with eight-point algorithm linear least-squares: unit norm vector F yielding smallest residual What happens when there is noise?

The Normalized Eight-Point Algorithm (Hartley, 1995) Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels: q = T p , q’ = T’ p’. Use the eight-point algorithm to compute F from the points q and q’ . Enforce the rank-2 constraint. Output T F T’. i i i i i i T

Epipolar geometry example

courtesy of Andrew Zisserman Example: converging cameras courtesy of Andrew Zisserman

Example: motion parallel with image plane (simple for stereo  rectification) courtesy of Andrew Zisserman

courtesy of Andrew Zisserman Example: forward motion e’ e courtesy of Andrew Zisserman

courtesy of Andrew Zisserman Fundamental matrix for pure translation auto-epipolar courtesy of Andrew Zisserman

courtesy of Andrew Zisserman Fundamental matrix for pure translation courtesy of Andrew Zisserman

Trinocular Epipolar Constraints These constraints are not independent!

Trinocular Epipolar Constraints: Transfer Given p and p , p can be computed as the solution of linear equations. 1 2 3

Trinocular Epipolar Constraints: Transfer problem for epipolar transfer in trifocal plane! There must be more to trifocal geometry… image from Hartley and Zisserman

Trifocal Constraints

Trifocal Constraints Calibrated Case All 3x3 minors must be zero! Trifocal Tensor

Trifocal Constraints Uncalibrated Case Trifocal Tensor

Trifocal Constraints: 3 Points Pick any two lines l and l through p and p . 2 3 2 3 Do it again. T( p , p , p )=0 1 2 3

For any matching epipolar lines, l G l = 0. Properties of the Trifocal Tensor For any matching epipolar lines, l G l = 0. The matrices G are singular. They satisfy 8 independent constraints in the uncalibrated case (Faugeras and Mourrain, 1995). T i 2 1 3 i 1 Estimating the Trifocal Tensor Ignore the non-linear constraints and use linear least-squares Impose the constraints a posteriori.

For any matching epipolar lines, l G l = 0. 2 1 3 The backprojections of the two lines do not define a line!

courtesy of Andrew Zisserman Trifocal Tensor Example 108 putative matches 18 outliers (26 samples) 88 inliers 95 final inliers (0.43) (0.23) (0.19) courtesy of Andrew Zisserman

Trifocal Tensor Example additional line matches images courtesy of Andrew Zisserman

Transfer: trifocal transfer (using tensor notation) doesn’t work if l’=epipolar line image courtesy of Hartley and Zisserman

Image warping using T(1,2,N) (Avidan and Shashua `97)

Multiple Views (Faugeras and Mourrain, 1995)

Two Views Epipolar Constraint

Three Views Trifocal Constraint

Four Views Quadrifocal Constraint (Triggs, 1995)

Geometrically, the four rays must intersect in P..

Quadrifocal Tensor and Lines

Quadrifocal tensor determinant is multilinear thus linear in coefficients of lines ! There must exist a tensor with 81 coefficients containing all possible combination of x,y,w coefficients for all 4 images: the quadrifocal tensor

Scale-Restraint Condition from Photogrammetry

Next class: Stereo (x´,y´)=(x+D(x,y),y) F&P Chapter 11 image I´(x´,y´) Disparity map D(x,y) image I´(x´,y´) (x´,y´)=(x+D(x,y),y) F&P Chapter 11