1. Computing F and rectification class 14 Multiple View Geometry Comp 290-089 Marc Pollefeys

2. Content Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation. Single View: Camera model, Calibration, Single View Geometry. Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies. Three Views: Trifocal Tensor, Computing T. More Views: N-Linearities, Multiple view reconstruction, Bundle adjustment, auto- calibration, Dynamic SfM, Cheirality, Duality

3. Multiple View Geometry course schedule (subject to change) Jan. 7, 9Intro & motivationProjective 2D Geometry Jan. 14, 16(no class)Projective 2D Geometry Jan. 21, 23Projective 3D Geometry(no class) Jan. 28, 30Parameter Estimation Feb. 4, 6Algorithm EvaluationCamera Models Feb. 11, 13Camera CalibrationSingle View Geometry Feb. 18, 20Epipolar Geometry3D reconstruction Feb. 25, 27Fund. Matrix Comp.Structure Comp. Mar. 4, 6Planes & HomographiesTrifocal Tensor Mar. 18, 20Three View ReconstructionMultiple View Geometry Mar. 25, 27MultipleView ReconstructionBundle adjustment Apr. 1, 3Auto-CalibrationPapers Apr. 8, 10Dynamic SfMPapers Apr. 15, 17CheiralityPapers Apr. 22, 24DualityProject Demos

4. Two-view geometry Epipolar geometry 3D reconstruction F-matrix comp. Structure comp.

5. Epipolar geometry: direct computation Basic equation 8-point algorithm (normalize!) 7-point algorithm (impose rank 2)

6. Epipolar geometry: iterative computation Maximum Likelihood Estimation (= least-squares for Gaussian noise) Sampson error (first order approx. to MLE) Symmetric epipolar error

7. Automatic computation of F (i)Interest points (ii)Putative correspondences (iii)RANSAC (iv) Non-linear re-estimation of F (v)Guided matching (repeat (iv) and (v) until stable)

8. Extract feature points to relate images Required properties: Well-defined (i.e. neigboring points should all be different) Stable across views (i.e. same 3D point should be extracted as feature for neighboring viewpoints) Feature points

9. homogeneous edge corner M should have large eigenvalues (e.g.Harris&Stephens´88; Shi&Tomasi´94) Find points that differ as much as possible from all neighboring points Feature = local maxima (subpixel) of F( 1, 2 ) Feature points

10. Select strongest features (e.g. 1000/image) Feature points

11. Evaluate NCC for all features with similar coordinates Keep mutual best matches Still many wrong matches! ? Feature matching

12. 0.96-0.40-0.16-0.390.19 -0.050.75-0.470.510.72 -0.18-0.390.730.15-0.75 -0.270.490.160.790.21 0.080.50-0.450.280.99 1 5 2 4 3 15 2 4 3 Gives satisfying results for small image motions Feature example

13. Requirement to cope with larger variations between images Translation, rotation, scaling Foreshortening Non-diffuse reflections Illumination geometric transformations photometric changes Wide-baseline matching…

14. Wide baseline matching for two different region types (Tuytelaars and Van Gool BMVC 2000) Wide-baseline matching…

15. Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do Step 3.1 select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F Step 3.3 determine inliers until  (#inliers,#samples)<95% #inliers90%80%70%60%50% #samples51335106382 Step 4. Compute F based on all inliers Step 5. Look for additional matches Step 6. Refine F based on all correct matches (generate hypothesis) (verify hypothesis) RANSAC

16. restrict search range to neighborhood of epipolar line (  1.5 pixels) relax disparity restriction (along epipolar line) Finding more matches

17. Degenerate cases Planar scene Pure rotation No unique solution Remaining DOF filled by noise Use simpler model (e.g. homography) Model selection (Torr et al., ICCV´98, Kanatani, Akaike) Compare H and F according to expected residual error (compensate for model complexity) Degenerate cases:

18. Model selection n = number of measurements (inliers+outliers) r = dimension of data k = motion model parameters d = dimension of structure structure motion MLE Geometric Robust Information Criterion

19. Model selection Video tracking H F Dominant planes

20. Absence of sufficient features (no texture) Repeated structure ambiguity (Schaffalitzky and Zisserman, BMVC‘98) Robust matcher also finds Robust matcher also finds support for wrong hypothesis support for wrong hypothesis solution: detect repetition solution: detect repetition More problems:

21. geometric relations between two views is fully described by recovered 3x3 matrix F two-view geometry

22. Image pair rectification simplify stereo matching by warping the images Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole e to (1,0,0) try to minimize image distortion problem when epipole in (or close to) the image

23. Planar rectification Bring two views to standard stereo setup (moves epipole to  ) (not possible when in/close to image) ~ image size (calibrated) Distortion minimization (uncalibrated) (standard approach)

26. Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose  so that no pixels are compressed original image rectified image Polar rectification (Pollefeys et al. ICCV’99) Works for all relative motions Guarantees minimal image size

27. polar rectification: example

29. Example: Béguinage of Leuven Does not work with standard Homography-based approaches

30. Example: Béguinage of Leuven

31. Exploiting motion and scene constraints Ordering constraint Uniqueness constraint Disparity limit Disparity continuity constraint Epipolar constraint Epipolar constraint (through rectification)

32. Ordering constraint 1 2 3 4,5 6 1 2,3 4 5 6 2 1 3 4,5 6 1 2,3 4 5 6 surface slice surface as a path occlusion right occlusion left

33. Uniqueness constraint In an image pair each pixel has at most one corresponding pixel In general one corresponding pixel In case of occlusion there is none

34. Disparity constraint surface slice surface as a path bounding box disparity band use reconsructed features to determine bounding box constant disparity surfaces

35. Disparity continuity constraint Assume piecewise continuous surface  piecewise continuous disparity In general disparity changes continuously discontinuities at occluding boundaries

36. Stereo matching Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints epipolar ordering uniqueness disparity limit disparity gradient limit Trade-off Matching cost (data) Discontinuities (prior) (Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

37. Hierarchical stereo matching Downsampling (Gaussian pyramid) Disparity propagation Allows faster computation Deals with large disparity ranges ( Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

38. Disparity map image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

39. Example: reconstruct image from neighboring images

40. Multi-view depth fusion Compute depth for every pixel of reference image Triangulation Use multiple views Up- and down sequence Use Kalman filter (Koch, Pollefeys and Van Gool. ECCV‘98) Allows to compute robust texture

41. Assignment 2 Compute F automatically from image pair (matches, 7-point, RANSAC, 8-point, iterative, more matches, epipolar lines, etc.) (due by Wednesday 19/03/03)

42. Next class: reconstructing points and lines