1. Multiple View Reconstruction Class 24 Multiple View Geometry Comp 290-089 Marc Pollefeys

2. Content Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation. Single View: Camera model, Calibration, Single View Geometry. Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies. Three Views: Trifocal Tensor, Computing T. More Views: N-Linearities, Self-Calibration,Multi View Reconstruction, Bundle adjustment, Dynamic SfM, Cheirality, Duality

3. Multi-view computation

4. practical structure and motion recovery from images Obtain reliable matches using matching or tracking and 2/3-view relations Compute initial structure and motion sequential structure and motion recovery hierarchical structure and motion recovery Refine structure and motion bundle adjustment Auto-calibrate Refine metric structure and motion

5. Sequential structure and motion recovery Initialize structure and motion from 2 views For each additional view Determine pose Refine and extend structure

6. Initial structure and motion Epipolar geometry  Projective calibration compatible with F Yields correct projective camera setup (Faugeras´92,Hartley´92) Obtain structure through triangulation Use reprojection error for minimization Avoid measurements in projective space

7. Compute P i+1 using robust approach (6-point RANSAC) Extend and refine reconstruction 2D-2D 2D-3D mimi m i+1 M new view Determine pose towards existing structure

8. Non-sequential image collections 4.8im/pt 64 images 3792 points Problem: Features are lost and reinitialized as new features Solution: Match with other close views

9. For every view i Extract features Compute two view geometry i-1/i and matches Compute pose using robust algorithm Refine existing structure Initialize new structure Relating to more views Problem: find close views in projective frame For every view i Extract features Compute two view geometry i-1/i and matches Compute pose using robust algorithm For all close views k Compute two view geometry k/i and matches Infer new 2D-3D matches and add to list Refine pose using all 2D-3D matches Refine existing structure Initialize new structure

10. Refining and extending structure Refining structure Extending structure Triangulation (Iterative linear) (Hartley&Sturm,CVIU´97) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

11. Structure and motion: example 190 images 7000points Input sequence Viewpoint surface mesh calibration demo

12. ULM demo

13. Hierarchical structure and motion recovery Compute 2-view Compute 3-view Stitch 3-view reconstructions Merge and refine reconstruction F T H PM

14. Stitching 3-view reconstructions Different possibilities 1. Align (P 2,P 3 ) with (P’ 1,P’ 2 ) 2. Align X,X’ (and C’C’) 3. Minimize reproj. error 4. MLE (merge)

15. Refining structure and motion Minimize reprojection error Maximum Likelyhood Estimation (if error zero-mean Gaussian noise) Huge problem but can be solved efficiently (Bundle adjustment)

16. Non-linear least-squares Newton iteration Levenberg-Marquardt Sparse Levenberg-Marquardt

17. Newton iteration Taylor approximation Jacobian normal eq.

18. Levenberg-Marquardt Augmented normal equations Normal equations solve again accept small ~ Newton (quadratic convergence) large ~ descent (guaranteed decrease)

19. Levenberg-Marquardt Requirements for minimization Function to compute f Start value P 0 Optionally, function to compute J (but numerical derivation ok too)

20. Sparse Levenberg-Marquardt complexity for solving prohibitive for large problems (100 views 10,000 points ~30,000 unknowns) Partition parameters partition A partition B (only dependent on A and itself) typically A contains camera parameters, and B contains 3D points

21. Sparse bundle adjustment residuals: normal equations: with

22. Sparse bundle adjustment normal equations: modified normal equations: solve in two parts:

23. Sparse bundle adjustment U1U1 U2U2 U3U3 WTWT W V P1P1 P2P2 P3P3 M Jacobian of has sparse block structure 12xm 3xn (in general much larger) im.pts. view 1 Needed for non-linear minimization

24. Sparse bundle adjustment Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m WTWT V U-WV -1 W T 11xm 3xn Allows much more efficient computations e.g. 100 views,10000 points, solve  1000x1000, not  30000x30000 Often still band diagonal use sparse linear algebra algorithms

25. Sparse bundle adjustment normal equations: modified normal equations: solve in two parts:

26. Sparse bundle adjustment Covariance estimation

27. Degenerate configurations Camera resectioning Two views More views (H&Z Ch.21)

28. Camera resectioning Cameras as points 2D case – Chasles’ theorem

29. Ambiguity for 3D cameras Twisted cubic (or less) meeting lin. subspace(s) (degree+dimension<3)

30. Ambiguous two-view reconstructions Ruled quadric containing both scene points and camera centers  alternative reconstructions exist for which the reconstruction of points located off the quadric are not projectively equivalent hyperboloid 1s cone pair of planes single plane + 2 points single line + 2 points

31. Multiple view reconstructions Single plane is still a problem Hartley and others looked at 3 and more view critical configurations, but those are rather exotic and are not a problem in practice.

32. Next class: Dynamic structure from motion