1. Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai

2. 2 Outline 3-D Reconstruction Fundamental matrix estimation Chapters 10 and 11 of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

3. 3 (i)Correspondence geometry: Given an image point x in the first image, how does this constrain the position of the corresponding point x’ in the second image? (ii)Camera geometry (motion): Given a set of corresponding image points {x i ↔x’ i }, i=1,…,n, what are the cameras P and P’ for the two views? (iii)Scene geometry (structure): Given corresponding image points x i ↔x’ i and cameras P, P’, what is the position of (their pre-image) X in space? Three questions:

4. 4 C1C1 C2C2 l2l2  l1l1 e1e1 e2e2 Fundamental matrix (3x3 rank 2 matrix) 1.Computable from corresponding points 2.Simplifies matching 3.Allows to detect wrong matches 4.Related to calibration Underlying structure in set of matches for rigid scenes l2l2 C1C1 m1m1 L1L1 m2m2 L2L2 M C2C2 m1m1 m2m2 C1C1 C2C2 l2l2  l1l1 e1e1 e2e2 m1m1 L1L1 m2m2 L2L2 M l2l2 lT1lT1 Canonical representation: Epipolar Geometry

5. 5 given x i ↔x‘ i, compute P,P‘ and X i reconstruction problem: for all i without additional information possible up to projective ambiguity 3D reconstruction of cameras and structure

6. 6 (i)Compute F from correspondences (ii)Compute camera matrices from F (iii)Compute 3D point for each pair of corresponding points computation of F use x‘ i Fx i =0 equations, linear in coeff. F 8 points (linear), 7 points (non-linear), 8+ (least-squares) (more on this next class) computation of camera matrices use triangulation compute intersection of two backprojected rays Outline of reconstruction

7. 7 Reconstruction ambiguity: similarity

8. 8 Reconstruction ambiguity: projective

9. 9 x i ↔x‘ i Original scene X i Projective, affine, similarity reconstruction = reconstruction that is identical to original up to projective, affine, similarity transformation Literature: Metric and Euclidean reconstruction = similarity reconstruction Terminology

10. 10 If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent  along same ray of P 2, idem for P ‘ 2 two possibilities: X 2 i = HX 1 i, or points along baseline key result: allows reconstruction from pair of uncalibrated images The projective reconstruction theorem

11. 11

12. 12 (i)Projective reconstruction (ii)Affine reconstruction (iii)Metric reconstruction Stratified reconstruction

13. 13 (if D ≠0) can be sufficient depending on application, e.g. mid-point, centroid, parallellism Projective to affine

14. 14 points at infinity (not necessarily visible) are fixed for a pure translation  reconstruction of x i ↔x i is on  ∞ Translational motion

15. 15 Parallel lines parallel lines intersect at infinity reconstruction of corresponding vanishing point yields point on plane at infinity 3 sets of parallel lines allow to uniquely determine  ∞ remark: in presence of noise determining the intersection of parallel lines is a delicate problem remark: obtaining vanishing point in one image can be sufficient Scene constraints

16. 16 Scene constraints

17. 17 identify absolute conic transform so that then projective transformation relating original and reconstruction is a similarity transformation in practice, find image of  ∞ image  ∞  back-projects to cone that intersects  ∞ in  ∞ ** ** projection constraints note that image is independent of particular reconstruction Affine to metric

18. 18 The absolute conic (reminder) The absolute conic Ω ∞ is a fixed conic under the projective transformation H iff H is a similarity The absolute conic Ω ∞ is a (point) conic on π . In a metric frame: or conic for directions: (with no real points) 1.Ω ∞ is only fixed as a set 2.Circles intersect Ω ∞ in two points 3.Spheres intersect π ∞ in Ω ∞

19. The image of the absolute conic It can be shown that the image of the absolute conic is: ω is a 3×3 symmetric matrix representing the conic: x T ωx=0 Even though the absolute conic contains only imaginary points, its image may include real points 19

20. 20 given possible transformation from affine to metric is (Cholesky factorization to obtain A) proof: Affine to metric

21. 21 vanishing points corresponding to orthogonal directions vanishing line and vanishing point corresponding to plane and normal direction Orthogonality

22. 22 rectangular pixels square pixels Known internal parameters

23. 23 same intrinsics  same image of the absolute conic e.g. moving camera given sufficient images there is in general only one conic that projects to the same image in all images, i.e. the absolute conic This approach is called self-calibration transfer of IAC: Same camera for all images provides 4 constraints, one more needed

24. 24 approach 1 calibrated reconstruction approach 2 compute projective reconstruction back-project  from both images intersection defines  ∞ and its support plane  ∞ (in general two solutions) Direct metric reconstruction using ω

25. 25

26. 26 use control points X E i with know coordinates to go from projective to metric (2 lin. eq. in H -1 per view, 3 for two views) Direct reconstruction using ground truth (3 lin. eq. in H per point, H has 15 d.o.f.)

27. 27 Given two uncalibrated images compute (P M,P‘ M,{X Mi }) (i.e. within similarity of original scene and cameras) Algorithm (i)Compute projective reconstruction (P,P‘,{X i }) (a)Compute F from x i ↔x‘ i (b)Compute P,P‘ from F (c)Triangulate X i from x i ↔x‘ i (ii)Rectify reconstruction from projective to metric Direct method: compute H from control points Stratified method: (a)Affine reconstruction: compute  ∞ (b)Metric reconstruction: compute IAC  Reconstruction summary

28. 28 Image information providedView relations and projective objects 3-space objects reconstruction ambiguity point correspondences F projective point correspondences including vanishing points F,H ∞ ∞∞ affine Points correspondences and internal camera calibration F,H ∞  ’ ∞∞∞∞ metric

29. 29 Outline 3-D Reconstruction Fundamental matrix estimation Chapters 10 and 11 of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

30. 30 separate known from unknown (data) (unknowns) (linear) Epipolar geometry: basic equation

31. 31 SVD from linearly computed F matrix (rank 3) Compute closest rank-2 approximation The singularity constraint

32. 32

33. 33 one parameter family of solutions but F 1 + F 2 not automatically rank 2 The minimum case – 7 point correspondences

34. 34 F1F1 F2F2 F 33 F 7pts (obtain 1 or 3 solutions) (cubic equation) Compute possible as eigenvalues of (only real solutions are potential solutions) The minimum case – impose rank 2

35. 35 ~10000 ~100 1 ! Orders of magnitude difference between column of data matrix  least-squares yields poor results The NOT normalized 8-point algorithm

36. 36 The normalized 8-point algorithm Transform image to ~[-1,1]x[-1,1] (0,0) (700,500) (700,0) (0,500) (1,-1) (0,0) (1,1)(-1,1) (-1,-1) normalized least squares yields good results (Hartley, PAMI´97)

37. 37 Gold standard Sampson error Symmetric epipolar distance Geometric distance

38. 38 Maximum Likelihood Estimation (= least-squares for Gaussian noise) Parameterize: Initialize: normalized 8-point, (P,P‘) from F, reconstruct X i Minimize cost using Levenberg-Marquardt (preferably sparse LM, see book) (overparametrized F=[t] x M) Gold standard

39. 39 Alternative, minimal parametrization (with a=1) (note (x,y,1) and (x‘,y‘,1) are epipoles) problems: a=0  pick largest of a,b,c,d to fix to 1 epipole at infinity  pick largest of x,y,w and of x’,y’,w’ 4x3x3=36 parametrizations! reparametrize at every iteration, to be sure Gold standard