Multiple View Reconstruction Class 24 Multiple View Geometry Comp Marc Pollefeys
Content Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation. Single View: Camera model, Calibration, Single View Geometry. Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies. Three Views: Trifocal Tensor, Computing T. More Views: N-Linearities, Self-Calibration,Multi View Reconstruction, Bundle adjustment, Dynamic SfM, Cheirality, Duality
Multi-view computation
practical structure and motion recovery from images Obtain reliable matches using matching or tracking and 2/3-view relations Compute initial structure and motion sequential structure and motion recovery hierarchical structure and motion recovery Refine structure and motion bundle adjustment Auto-calibrate Refine metric structure and motion
Sequential structure and motion recovery Initialize structure and motion from 2 views For each additional view Determine pose Refine and extend structure
Initial structure and motion Epipolar geometry Projective calibration compatible with F Yields correct projective camera setup (Faugeras´92,Hartley´92) Obtain structure through triangulation Use reprojection error for minimization Avoid measurements in projective space
Compute P i+1 using robust approach (6-point RANSAC) Extend and refine reconstruction 2D-2D 2D-3D mimi m i+1 M new view Determine pose towards existing structure
Non-sequential image collections 4.8im/pt 64 images 3792 points Problem: Features are lost and reinitialized as new features Solution: Match with other close views
For every view i Extract features Compute two view geometry i-1/i and matches Compute pose using robust algorithm Refine existing structure Initialize new structure Relating to more views Problem: find close views in projective frame For every view i Extract features Compute two view geometry i-1/i and matches Compute pose using robust algorithm For all close views k Compute two view geometry k/i and matches Infer new 2D-3D matches and add to list Refine pose using all 2D-3D matches Refine existing structure Initialize new structure
Refining and extending structure Refining structure Extending structure Triangulation (Iterative linear) (Hartley&Sturm,CVIU´97) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion
Structure and motion: example 190 images 7000points Input sequence Viewpoint surface mesh calibration demo
ULM demo
Hierarchical structure and motion recovery Compute 2-view Compute 3-view Stitch 3-view reconstructions Merge and refine reconstruction F T H PM
Stitching 3-view reconstructions Different possibilities 1. Align (P 2,P 3 ) with (P’ 1,P’ 2 ) 2. Align X,X’ (and C’C’) 3. Minimize reproj. error 4. MLE (merge)
Refining structure and motion Minimize reprojection error Maximum Likelyhood Estimation (if error zero-mean Gaussian noise) Huge problem but can be solved efficiently (Bundle adjustment)
Non-linear least-squares Newton iteration Levenberg-Marquardt Sparse Levenberg-Marquardt
Newton iteration Taylor approximation Jacobian normal eq.
Levenberg-Marquardt Augmented normal equations Normal equations solve again accept small ~ Newton (quadratic convergence) large ~ descent (guaranteed decrease)
Levenberg-Marquardt Requirements for minimization Function to compute f Start value P 0 Optionally, function to compute J (but numerical derivation ok too)
Sparse Levenberg-Marquardt complexity for solving prohibitive for large problems (100 views 10,000 points ~30,000 unknowns) Partition parameters partition A partition B (only dependent on A and itself) typically A contains camera parameters, and B contains 3D points
Sparse bundle adjustment residuals: normal equations: with
Sparse bundle adjustment normal equations: modified normal equations: solve in two parts:
Sparse bundle adjustment U1U1 U2U2 U3U3 WTWT W V P1P1 P2P2 P3P3 M Jacobian of has sparse block structure 12xm 3xn (in general much larger) im.pts. view 1 Needed for non-linear minimization
Sparse bundle adjustment Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m WTWT V U-WV -1 W T 11xm 3xn Allows much more efficient computations e.g. 100 views,10000 points, solve 1000x1000, not 30000x30000 Often still band diagonal use sparse linear algebra algorithms
Sparse bundle adjustment normal equations: modified normal equations: solve in two parts:
Sparse bundle adjustment Covariance estimation
Degenerate configurations Camera resectioning Two views More views (H&Z Ch.21)
Camera resectioning Cameras as points 2D case – Chasles’ theorem
Ambiguity for 3D cameras Twisted cubic (or less) meeting lin. subspace(s) (degree+dimension<3)
Ambiguous two-view reconstructions Ruled quadric containing both scene points and camera centers alternative reconstructions exist for which the reconstruction of points located off the quadric are not projectively equivalent hyperboloid 1s cone pair of planes single plane + 2 points single line + 2 points
Multiple view reconstructions Single plane is still a problem Hartley and others looked at 3 and more view critical configurations, but those are rather exotic and are not a problem in practice.
Next class: Dynamic structure from motion