1. Stereo Course web page: vision.cis.udel.edu/~cv April 11, 2003  Lecture 21

2. Announcements No class on Monday Read Forsyth & Ponce, Chapter 11-11.1 and Hartley & Zisserman, Chapter 10- 10.2 on calculating the fundamental matrix for next Wednesday HW4 will be assigned next Wednesday and due Monday, April 28

3. Outline Epipolar geometry Fundamental matrix Depth recovery

4. Two-View Geometry The relationship of two views of a scene taken from different camera positions to one another Interpretations –“Stereo vision” generally means two synchronized cameras or eyes capturing images –Could also be two sequential views from the same camera in motion Assuming a static scene

5. Mapping Points between Images What is the relationship between the images x, x’ of the scene point X in two views? Intuitively, it depends on: –The rigid transformation between cameras (derivable from the camera matrices P, P’ ) –The scene structure (i.e., the depth of X ) Parallax: Closer points appear to move more

6. Example: Two-View Geometry courtesy of F. Dellaert x1x1 x’1x’1 x2x2 x’2x’2 x3x3 x’3x’3 Is there a transformation relating the points x i to x’ i ?

7. Epipolar Geometry Baseline: Line joining camera centers C, C’ Epipolar plane ¦ : Defined by baseline and scene point X from Hartley & Zisserman baseline

8. Epipolar Lines Epipolar lines l, l’ : Intersection of epipolar plane ¦ with image planes Epipoles e, e’ : Where baseline intersects image planes –Equivalently, the image in one view of the other camera center. C C’C’ from Hartley & Zisserman

9. Epipolar Pencil As position of X varies, epipolar planes “rotate” about the baseline –This set of planes is called the epipolar pencil Epipolar lines “radiate” from epipole—this is the pencil of epipolar lines from Hartley & Zisserman

10. Epipolar Constraint Camera center C and image point define ray in 3-D space that projects to epipolar line l’ in other view (since it’s on the epipolar plane) 3-D point X on this ray, so image of X in other view x’ must be on l’ In other words, the epipolar geometry defines a mapping x ! l’ of points in one image to lines in the other from Hartley & Zisserman C C’C’ x’x’

11. Example: Epipolar Lines for Converging Cameras from Hartley & Zisserman Left viewRight view Intersection of epipolar lines = Epipole ! Indicates location of other camera center

12. Special Case: Translation Parallel to Image Plane Note that epipolar lines are parallel and corresponding points lie on correspond- ing epipolar lines (the latter is true for all kinds of camera motions)

13. Special Case: Translation along Optical Axis Epipoles coincide at focus of expansion Not the same (in general) as vanishing point of scene lines from Hartley & Zisserman

14. The Fundamental Matrix F Mapping of point in one image to epipolar line in other image x ! l’ is expressed algebraically by the fundamental matrix F Write this as l’ = F x Remember that the point-on-line relationship is given by l’ ¢ Fx = 0, which can also be written as l’ T Fx = (Fx) T l’ = 0 F is 3 x 3, rank 2 (not invertible, in contrast to homographies) –7 DOF (homogeneity and rank constraint take away 2 DOF) line point

15. The Fundamental Matrix F Since x’ is on l’, by the point-on-line definition we know that x’ T l’ = 0 Combined with l’ = F x, we can thus relate corres- ponding points in the camera pair (P, P’) to each other with the following: x’ T F x = 0 The fundamental matrix of (P’, P) is the transpose F T from Hartley & Zisserman x’x’

16. Deducing Epipoles from F For any x (besides e ), l’ = F x contains e’, so e’ T (F x) = 0. Thus, e’ T F = 0 ! F T e’ = 0 –Similarly, Fe = 0 Solve systems Fe = 0, F T e’ = 0 –Same as last step of DLT (recall that there we were trying to solve Ap = 0 ) –E.g., to solve Fe = 0 in Matlab: Take SVD of F with [U,S,V] = svd(F) Last column of V is the answer e x’x’ from Hartley & Zisserman

17. Cross Products & Skew- Symmetric Matrices If a = (a 1, a 2, a 3 ), then define With this we have the following identity:

18. Building F from Known Camera Matrices P, P’ P, P’ determine unique fundamental matrix by F = [e’] £ P’P + –Pseudoinverse A + ( pinv in Matlab): Like inverse for non-square matrices (i.e., AA + A = A, etc.) Defined as A + = (A T A) -1 A T when inverse exists Can also get camera matrices from F –Canonical form: P = K[Id j 0], P’ = K’[R j t] –However, there is an ambiguity up to a projective transformation H such that we actually get PH, P’H

19. The Essential Matrix E Fundamental matrix when calibration matrices K, K’ of the two cameras are known is called the essential matrix E : Allows computation of camera matrices P, P’ up to a scale ambiguity

20. Extracting Structure The key aspect of epipolar geometry is its linear constraint on where a point in one image can be in the other By correlation-matching pixels (or features) along epipolar lines and measuring the disparity between them, we can construct a depth map (scene point depth is inversely proportional to disparity) View 1View 2Computed depth map courtesy of P. Debevec