1. Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai

2. 2 Outline Fundamental matrix estimation Image rectification Structure Computation Stereo Chapter 11 and 12of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

3. 3 (i)Interest points (ii)Putative correspondences (iii)RANSAC (iv) Non-linear re-estimation of F (v)Guided matching (repeat (iv) and (v) until stable) Automatic computation of F

4. 4 Feature points Extract feature points to relate images Required properties: –Well-defined (i.e. neigboring points should all be different) –Stable across views (i.e. same 3D point should be extracted as feature for neighboring viewpoints)

5. 5 homogeneous edge corner M should have large eigenvalues (e.g.Harris&Stephens´88; Shi&Tomasi´94) Find points that differ as much as possible from all neighboring points Feature = local maxima (subpixel) of F( 1, 2 ) Feature points

6. 6 Select strongest features (e.g. 1000/image) Feature points

7. 7 Feature matching Evaluate NCC for all features with similar coordinates Keep mutual best matches Still many wrong matches! ?

8. 8 0.96-0.40-0.16-0.390.19 -0.050.75-0.470.510.72 -0.18-0.390.730.15-0.75 -0.270.490.160.790.21 0.080.50-0.450.280.99 1 5 2 4 3 15 2 4 3 Gives satisfying results for small image motions Feature example

9. 9 Wide-baseline matching Requirement to cope with larger variations between images –Translation, rotation, scaling –Foreshortening –Non-diffuse reflections –Illumination geometric transformations photometric changes

10. 10 RANSAC Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F Step 3.3 determine inliers until  (#inliers,#samples)<95% #inliers90%80%70%60%50% #samples51335106382 Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches (generate hypothesis) (verify hypothesis)

11. 11 restrict search range to neighborhood of epipolar line (  1.5 pixels) relax disparity restriction (along epipolar line) Finding more matches

12. 12 Degenerate cases: Degenerate cases –Planar scene –Pure rotation No unique solution –Remaining DOF filled by noise –Use simpler model (e.g. homography) Model selection (Torr et al., ICCV´98, Kanatani, Akaike) –Compare H and F according to expected residual error (compensate for model complexity)

13. 13 More problems: Absence of sufficient features (no texture) Repeated structure ambiguity (Schaffalitzky and Zisserman, BMVC‘98) Robust matcher also finds Robust matcher also finds support for wrong hypothesis support for wrong hypothesis solution: detect repetition solution: detect repetition

14. 14 Two-view geometry geometric relation between two views is fully described by recovered 3x3 matrix F

15. 15 Outline Fundamental matrix estimation Image rectification Structure Computation Stereo Chapter 11 and 12of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

16. 16 Point reconstruction

17. 17 homogeneous inhomogeneous invariance? Inhomogeneous is affine invariant Linear triangulation

18. 18 Can be compute using Levenberg-Marquadt (for 2 or more points) or directly by solving 6 th degree polynomial (for 2 points) Geometric error

19. 19 Reconstruction uncertainty consider angle between rays

20. 20 Outline Fundamental matrix estimation Image rectification Structure Computation Stereo Chapter 11 and 12of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

21. 21 simplify stereo matching by warping the images Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole e to (1,0,0) try to minimize image distortion problem when epipole in (or close to) the image Image pair rectification

22. 22 Planar rectification Bring two views to standard stereo setup (moves epipole to  ) (not possible when in/close to image) ~ image size (calibrated) Distortion minimization (uncalibrated) (standard approach)

23. 23

24. 24

25. 25 Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose  so that no pixels are compressed original image rectified image Polar rectification (Pollefeys et al. ICCV’99) Works for all relative motions Guarantees minimal image size

26. 26 polar rectification: example

27. 27 polar rectification: example

28. 28 Example: Béguinage of Leuven Does not work with standard Homography-based approaches

29. 29 Example: Béguinage of Leuven

30. 30 Outline Fundamental matrix estimation Image rectification Structure Computation Stereo Chapter 11 and 12of “Multiple View Geometry in Computer Vision” by Hartley and Zisserman

31. 31 Stereo matching attempt to match every pixel use additional constraints

32. 32 Exploiting motion and scene constraints Ordering constraint Uniqueness constraint Disparity continuity constraint Epipolar constraint Epipolar constraint (through rectification)

33. 33 Ordering constraint 1 2 3 4,5 6 1 2,3 4 5 6 2 1 3 4,5 6 1 2,3 4 5 6 surface slice surface as a path occlusion right occlusion left

34. 34 Uniqueness constraint In an image pair each pixel has at most one corresponding pixel –In general one corresponding pixel –In case of occlusion there is none

35. 35 Disparity continuity constraint Assume piecewise continuous surface  piecewise continuous disparity –In general disparity changes continuously –discontinuities at occluding boundaries

36. 36 Stereo matching Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints epipolar ordering uniqueness disparity limit disparity gradient limit Trade-off Matching cost (data) Discontinuities (prior) (Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

37. 37 Disparity map image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

38. 38 Hierarchical stereo matching Downsampling (Gaussian pyramid) Disparity propagation Allows faster computation Deals with large disparity ranges ( Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)