1. Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai

2. 2 Outline Fundamental matrix estimation Image rectification Chapter 10 of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

3. 3 separate known from unknown (data) (unknowns) (linear) Epipolar geometry: basic equation

4. 4 SVD from linearly computed F matrix (rank 3) Compute closest rank-2 approximation The singularity constraint

5. 5

6. 6 one parameter family of solutions but F 1 + F 2 not automatically rank 2 The minimum case – 7 point correspondences

7. 7 F1F1 F2F2 F 33 F 7pts (obtain 1 or 3 solutions) (cubic equation) Compute possible as eigenvalues of (only real solutions are potential solutions) The minimum case – impose rank 2

8. 8 ~10000 ~100 1 ! Orders of magnitude difference between column of data matrix  least-squares yields poor results The NOT normalized 8-point algorithm

9. 9 The normalized 8-point algorithm Transform image to ~[-1,1]x[-1,1] (0,0) (700,500) (700,0) (0,500) (1,-1) (0,0) (1,1)(-1,1) (-1,-1) normalized least squares yields good results (Hartley, PAMI´97)

10. 10 Gold standard Sampson error Symmetric epipolar distance Geometric distance

11. 11 Maximum Likelihood Estimation (= least-squares for Gaussian noise) Parameterize: Initialize: normalized 8-point, (P,P‘) from F, reconstruct X i Minimize cost using Levenberg-Marquardt (preferably sparse LM, see book) (overparametrized F=[t] x M) Gold standard

12. 12 Alternative, minimal parametrization (with a=1) (note (x,y,1) and (x‘,y‘,1) are epipoles) problems: a=0  pick largest of a,b,c,d to fix to 1 epipole at infinity  pick largest of x,y,w and of x’,y’,w’ 4x3x3=36 parametrizations! reparametrize at every iteration, to be sure Gold standard

13. 13 (one eq./point  JJ T scalar) (problem if some x is located at epipole) advantage: no subsidiary variables required (coordinates of world points) First-order geometric error (Sampson error)

14. 14 Symmetric epipolar error

15. 15 Some experiments:

16. 16 Some experiments:

17. 17 Some experiments:

18. 18 Some experiments: (for all matches, not just n) Residual error: Solid: 8-point algorithm Dashed: geometric error Dotted: algebraic error

19. 19 1.Do not use unnormalized algorithms 2.Quick and easy to implement: 8-point normalized 3.Better: enforce rank-2 constraint during minimization 4.Best: Maximum Likelihood Estimation (minimal parameterization, sparse implementation) Recommendations:

20. 20 Enforce constraints for optimal results: Pure translation (2dof), Planar motion (6dof), Calibrated case (5dof) – Essential matrix Special cases:

21. 21 What happens to an epipolar line if there is noise? Monte Carlo n =10 n =15 n =25 n =50 The envelope of epipolar lines Notice that narrower part of envelope is not the correct match

22. 22 Other entities? Lines give no constraint for two view geometry (but will for three and more views) Curves and surfaces yield some constraints related to tangency

23. 23 (i)Interest points (ii)Putative correspondences (iii)RANSAC (iv) Non-linear re-estimation of F (v)Guided matching (repeat (iv) and (v) until stable) Automatic computation of F

24. 24 Feature points Extract feature points to relate images Required properties: –Well-defined (i.e. neigboring points should all be different) –Stable across views (i.e. same 3D point should be extracted as feature for neighboring viewpoints)

25. 25 homogeneous edge corner M should have large eigenvalues (e.g.Harris&Stephens´88; Shi&Tomasi´94) Find points that differ as much as possible from all neighboring points Feature = local maxima (subpixel) of F( 1, 2 ) Feature points

26. 26 Select strongest features (e.g. 1000/image) Feature points

27. 27 Feature matching Evaluate NCC for all features with similar coordinates Keep mutual best matches Still many wrong matches! ?

28. 28 0.96-0.40-0.16-0.390.19 -0.050.75-0.470.510.72 -0.18-0.390.730.15-0.75 -0.270.490.160.790.21 0.080.50-0.450.280.99 1 5 2 4 3 15 2 4 3 Gives satisfying results for small image motions Feature example

29. 29 Wide-baseline matching Requirement to cope with larger variations between images –Translation, rotation, scaling –Foreshortening –Non-diffuse reflections –Illumination geometric transformations photometric changes

30. 30 RANSAC Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F Step 3.3 determine inliers until  (#inliers,#samples)<95% #inliers90%80%70%60%50% #samples51335106382 Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches (generate hypothesis) (verify hypothesis)

31. 31 restrict search range to neighborhood of epipolar line (  1.5 pixels) relax disparity restriction (along epipolar line) Finding more matches

32. 32 Degenerate cases: Degenerate cases –Planar scene –Pure rotation No unique solution –Remaining DOF filled by noise –Use simpler model (e.g. homography) Model selection (Torr et al., ICCV´98, Kanatani, Akaike) –Compare H and F according to expected residual error (compensate for model complexity)

33. 33 More problems: Absence of sufficient features (no texture) Repeated structure ambiguity (Schaffalitzky and Zisserman, BMVC‘98) Robust matcher also finds Robust matcher also finds support for wrong hypothesis support for wrong hypothesis solution: detect repetition solution: detect repetition

34. 34 Two-view geometry geometric relation between two views is fully described by recovered 3x3 matrix F

35. 35 Outline Fundamental matrix estimation Image rectification Chapter 10 of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

36. 36 simplify stereo matching by warping the images Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole e to (1,0,0) try to minimize image distortion problem when epipole in (or close to) the image Image pair rectification

37. 37 Planar rectification Bring two views to standard stereo setup (moves epipole to  ) (not possible when in/close to image) ~ image size (calibrated) Distortion minimization (uncalibrated) (standard approach)

38. 38

39. 39

40. 40 Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose  so that no pixels are compressed original image rectified image Polar rectification (Pollefeys et al. ICCV’99) Works for all relative motions Guarantees minimal image size

41. 41 polar rectification: example

42. 42 polar rectification: example

43. 43 Example: Béguinage of Leuven Does not work with standard Homography-based approaches

44. 44 Example: Béguinage of Leuven