1. Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai

2. 2 Outline Computation of camera matrix P Epipolar geometry estimation Chapters 6 and 8 of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

3. 3 Projection Matrix Estimation Similar to H estimation

4. 4 minimal solution Over-determined solution  5½ correspondences needed (say 6) P has 11 dof, 2 independent eq./points n  6 points minimize subject to constraint Basic equations

5. 5 Scaling Data normalization Centroid at origin

6. 6 Gold Standard algorithm Objective Given n≥6 3D to 2D point correspondences {X i ↔x i ’}, determine the Maximum Likelihood Estimation of P Algorithm (i)Linear solution: (a)Normalization: (b)DLT (ii)Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error: (iii)Denormalization: ~~ ~

7. 7 (i)Canny edge detection (ii)Straight line fitting to the detected edges (iii)Intersecting the lines to obtain the images corners typically precision <1/10 (HZ rule of thumb: 5 n constraints for n unknowns) Calibration example

8. 8 Errors in the image and in the world Errors in the world

9. 9 Minimize geometric error  impose constraint through parameterization  Image only  9   2n, otherwise  3n+9   5n Find best fit that satisfies skew s is zero pixels are square principal point is known complete camera matrix K is known Minimize algebraic error  assume map from param q  P=K[R|-RC], i.e. p=g(q)  minimize ||Ag(q)|| (9 instead of 12 parameters) Restricted camera estimation

10. 10 One only has to work with 12x12 matrix, not 2nx12 Optimization cost is independent of the number of correspondences Reduced measurement matrix

11. 11 Initialization Use general DLT Clamp values to desired values, e.g. s=0,  x =  y Note: can sometimes cause big jump in error Alternative initialization Use general DLT Impose soft constraints gradually increase weights Restricted camera estimation

12. 12 ML residual error Example: n=197, =0.365, =0.37 Covariance estimation d: number of parameters

13. 13 Compute Jacobian of measured points in terms of camera parameters at ML solution, then (variance per parameter can be found on diagonal) (chi-square distribution =distribution of sum of squares) cumulative -1 Covariance for estimated camera Confidence ellipsoid for camera center:

14. 14

15. 15 Outline Computation of camera matrix P Epipolar geometry estimation

16. 16 (i)Correspondence geometry: Given an image point x in the first image, how does this constrain the position of the corresponding point x’ in the second image? (ii)Camera geometry (motion): Given a set of corresponding image points {x i ↔x’ i }, i=1,…,n, what are the cameras P and P’ for the two views? (iii)Scene geometry (structure): Given corresponding image points x i ↔x’ i and cameras P, P’, what is the position of (their pre-image) X in space? Three questions:

17. 17 C,C’,x,x’ and X are coplanar The epipolar geometry

18. 18 What if only C,C’,x are known? The epipolar geometry

19. 19 All points on  project on l and l’ The epipolar geometry

20. 20 Family of planes  and lines l and l’ Intersection in e and e’ The epipolar geometry

21. 21 epipoles e,e’ = intersection of baseline with image plane = projection of projection center in other image = vanishing point of camera motion direction an epipolar plane = plane containing baseline (1-D family) an epipolar line = intersection of epipolar plane with image (always come in corresponding pairs) The epipolar geometry

22. 22 Example: converging cameras

23. 23 (simple for stereo  rectification) Example: motion parallel with image plane

24. 24 e e’ Example: forward motion

25. 25 The fundamental matrix F algebraic representation of epipolar geometry we will see that mapping is a (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F

26. 26 The fundamental matrix F geometric derivation mapping from 2-D to 1-D family (rank 2)

27. 27 The fundamental matrix F (note: doesn’t work for C=C’  F=0)

28. 28 The fundamental matrix F correspondence condition The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x ’ in the two images

29. 29 The fundamental matrix F F is the unique 3x3 rank 2 matrix that satisfies x’ T Fx=0 for all x↔x’ (i)Transpose: if F is fundamental matrix for (P,P’), then F T is fundamental matrix for (P’,P) (ii)Epipolar lines: l’=Fx & l=F T x’ (iii)Epipoles: on all epipolar lines, thus e’ T Fx=0,  x  e’ T F=0, similarly Fe=0 (iv)F has 7 d.o.f., i.e. 3x3-1(homogeneous)-1(rank2) (v)F is a correlation, projective mapping from a point x to a line l’=Fx (not a proper correlation, i.e. not invertible)

30. 30 Fundamental matrix for pure translation

31. 31 Fundamental matrix for pure translation

32. 32 Fundamental matrix for pure translation example: motion starts at x and moves towards e, faster depending on Z pure translation: F only 2 d.o.f., x T [e] x x=0  auto-epipolar e x x’ y’ y

33. 33 General motion

34. 34 Projective transformation and invariance Derivation based purely on projective concepts F invariant to transformations of projective 3-space unique not unique canonical form

35. 35 Projective ambiguity of cameras given F previous slide: at least projective ambiguity this slide: not more! Show that if F is same for (P,P’) and (P,P’), there exists a projective transformation H so that P=HP and P’=HP’ ~ ~ ~ lemma: (22-15=7, ok)

36. 36 The essential matrix ~fundamental matrix for calibrated cameras (remove K) 5 d.o.f. (3 for R; 2 for t up to scale) E is essential matrix if and only if two singular values are equal (and third=0)

37. 37 Four possible reconstructions from E (only one solution where points is in front of both cameras)

38. 38 Next week: 3D reconstruction

39. 39 Reconstruction problem given x i ↔x‘ i, compute P,P‘ and X i for all i without additional information possible up to projective ambiguity

40. 40 Outline of reconstruction (i)Compute F from correspondences (ii)Compute camera matrices from F (iii)Compute 3D point for each pair of corresponding points computation of F use x‘ i Fx i =0 equations, linear in coeff. F 8 points (linear), 7 points (non-linear), 8+ (least-squares) (more on this next class) computation of camera matrices use triangulation compute intersection of two backprojected rays