Multiple View Reconstruction Class 24 Multiple View Geometry Comp 290-089 Marc Pollefeys.

Slides:



Advertisements
Similar presentations
Projective 3D geometry class 4
Advertisements

The fundamental matrix F
The Trifocal Tensor Multiple View Geometry. Scene planes and homographies plane induces homography between two views.
Computing 3-view Geometry Class 18
MASKS © 2004 Invitation to 3D vision Lecture 7 Step-by-Step Model Buidling.
Two-View Geometry CS Sastry and Yang
N-view factorization and bundle adjustment CMPUT 613.
Structure from motion Class 9 Read Chapter 5. 3D photography course schedule (tentative) LectureExercise Sept 26Introduction- Oct. 3Geometry & Camera.
Epipolar Geometry class 11 Multiple View Geometry Comp Marc Pollefeys.
Structure from motion.
Scene Planes and Homographies class 16 Multiple View Geometry Comp Marc Pollefeys.
Projective structure from motion
Multiple View Geometry
More on single-view geometry class 10 Multiple View Geometry Comp Marc Pollefeys.
Robot Vision SS 2008 Matthias Rüther 1 ROBOT VISION Lesson 6: Shape from Stereo Matthias Rüther Slides partial courtesy of Marc Pollefeys Department of.
Computer Vision cmput 613 Sequential 3D Modeling from images using epipolar geometry and F 3D Modeling from images using epipolar geometry and F Martin.
3D reconstruction class 11
Projective 2D geometry (cont’) course 3
Parameter estimation class 5 Multiple View Geometry Comp Marc Pollefeys.
Computer Vision Projective structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, …
Computing F and rectification class 14 Multiple View Geometry Comp Marc Pollefeys.
Epipolar geometry. (i)Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point.
Structure from motion. Multiple-view geometry questions Scene geometry (structure): Given 2D point matches in two or more images, where are the corresponding.
Parameter estimation class 6 Multiple View Geometry Comp Marc Pollefeys.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Many slides and illustrations from J. Ponce
Self-calibration Class 21 Multiple View Geometry Comp Marc Pollefeys.
Multiple View Geometry Comp Marc Pollefeys
Lecture 11: Structure from motion CS6670: Computer Vision Noah Snavely.
Two-view geometry Epipolar geometry F-matrix comp. 3D reconstruction Structure comp.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Self-calibration Class 13 Read Chapter 6. Assignment 3 Collect potential matches from all algorithms for all pairs Matlab ASCII format, exchange data.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Triangulation and Multi-View Geometry Class 9 Read notes Section 3.3, , 5.1 (if interested, read Triggs’s paper on MVG using tensor notation, see.
Multiple View Geometry in Computer Vision
Assignment 2 Compute F automatically from image pair (putative matches, 8-point, 7-point, iterative, RANSAC, guided matching) (due by Wednesday 19/03/03)
 -Linearities and Multiple View Tensors Class 19 Multiple View Geometry Comp Marc Pollefeys.
Structure from motion Class 12 Read Chapter 5. Assignment 2 ChrisMS regions Nathan… BrianM&S LoG features LiSIFT features ChadMS regions Seon JooSIFT.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
More on single-view geometry class 10 Multiple View Geometry Comp Marc Pollefeys.
Multiple View Reconstruction Class 23 Multiple View Geometry Comp Marc Pollefeys.
Algorithm Evaluation and Error Analysis class 7 Multiple View Geometry Comp Marc Pollefeys.
Multiple View Geometry Marc Pollefeys University of North Carolina at Chapel Hill Modified by Philippos Mordohai.
Camera Calibration class 9 Multiple View Geometry Comp Marc Pollefeys.
Global Alignment and Structure from Motion Computer Vision CSE455, Winter 2008 Noah Snavely.
Computer Vision Calibration Marc Pollefeys COMP 256 Read F&P Chapter 2 Some slides/illustrations from Ponce, Hartley & Zisserman.
The Trifocal Tensor Class 17 Multiple View Geometry Comp Marc Pollefeys.
Epipolar geometry Class 5. Geometric Computer Vision course schedule (tentative) LectureExercise Sept 16Introduction- Sept 23Geometry & Camera modelCamera.
Multiple View Geometry in Computer Vision Slides modified from Marc Pollefeys’ online course materials Lecturer: Prof. Dezhen Song.
Geometry and Algebra of Multiple Views
Camera Calibration class 9 Multiple View Geometry Comp Marc Pollefeys.
Projective cameras Motivation Elements of Projective Geometry Projective structure from motion Planches : –
Robot Vision SS 2007 Matthias Rüther 1 ROBOT VISION Lesson 6a: Shape from Stereo, short summary Matthias Rüther Slides partial courtesy of Marc Pollefeys.
Parameter estimation. 2D homography Given a set of (x i,x i ’), compute H (x i ’=Hx i ) 3D to 2D camera projection Given a set of (X i,x i ), compute.
Communication Systems Group Technische Universität Berlin S. Knorr A Geometric Segmentation Approach for the 3D Reconstruction of Dynamic Scenes in 2D.
Raquel A. Romano 1 Scientific Computing Seminar May 12, 2004 Projective Geometry for Computer Vision Projective Geometry for Computer Vision Raquel A.
EECS 274 Computer Vision Geometric Camera Calibration.
3D reconstruction from uncalibrated images
Structure from motion Multi-view geometry Affine structure from motion Projective structure from motion Planches : –
EECS 274 Computer Vision Projective Structure from Motion.
Parameter estimation class 5 Multiple View Geometry CPSC 689 Slides modified from Marc Pollefeys’ Comp
Parameter estimation class 5
Epipolar Geometry class 11
3D Photography: Epipolar geometry
Multiple View Geometry Comp Marc Pollefeys
More on single-view geometry class 10
Estimating 2-view relationships
3D reconstruction class 11
Camera Calibration class 9
Parameter estimation class 6
Presentation transcript:

Multiple View Reconstruction Class 24 Multiple View Geometry Comp Marc Pollefeys

Content Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation. Single View: Camera model, Calibration, Single View Geometry. Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies. Three Views: Trifocal Tensor, Computing T. More Views: N-Linearities, Self-Calibration,Multi View Reconstruction, Bundle adjustment, Dynamic SfM, Cheirality, Duality

Multi-view computation

practical structure and motion recovery from images Obtain reliable matches using matching or tracking and 2/3-view relations Compute initial structure and motion sequential structure and motion recovery hierarchical structure and motion recovery Refine structure and motion bundle adjustment Auto-calibrate Refine metric structure and motion

Sequential structure and motion recovery Initialize structure and motion from 2 views For each additional view Determine pose Refine and extend structure

Initial structure and motion Epipolar geometry  Projective calibration compatible with F Yields correct projective camera setup (Faugeras´92,Hartley´92) Obtain structure through triangulation Use reprojection error for minimization Avoid measurements in projective space

Compute P i+1 using robust approach (6-point RANSAC) Extend and refine reconstruction 2D-2D 2D-3D mimi m i+1 M new view Determine pose towards existing structure

Non-sequential image collections 4.8im/pt 64 images 3792 points Problem: Features are lost and reinitialized as new features Solution: Match with other close views

For every view i Extract features Compute two view geometry i-1/i and matches Compute pose using robust algorithm Refine existing structure Initialize new structure Relating to more views Problem: find close views in projective frame For every view i Extract features Compute two view geometry i-1/i and matches Compute pose using robust algorithm For all close views k Compute two view geometry k/i and matches Infer new 2D-3D matches and add to list Refine pose using all 2D-3D matches Refine existing structure Initialize new structure

Refining and extending structure Refining structure Extending structure Triangulation (Iterative linear) (Hartley&Sturm,CVIU´97) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

Structure and motion: example 190 images 7000points Input sequence Viewpoint surface mesh calibration demo

ULM demo

Hierarchical structure and motion recovery Compute 2-view Compute 3-view Stitch 3-view reconstructions Merge and refine reconstruction F T H PM

Stitching 3-view reconstructions Different possibilities 1. Align (P 2,P 3 ) with (P’ 1,P’ 2 ) 2. Align X,X’ (and C’C’) 3. Minimize reproj. error 4. MLE (merge)

Refining structure and motion Minimize reprojection error Maximum Likelyhood Estimation (if error zero-mean Gaussian noise) Huge problem but can be solved efficiently (Bundle adjustment)

Non-linear least-squares Newton iteration Levenberg-Marquardt Sparse Levenberg-Marquardt

Newton iteration Taylor approximation Jacobian normal eq.

Levenberg-Marquardt Augmented normal equations Normal equations solve again accept small ~ Newton (quadratic convergence) large ~ descent (guaranteed decrease)

Levenberg-Marquardt Requirements for minimization Function to compute f Start value P 0 Optionally, function to compute J (but numerical derivation ok too)

Sparse Levenberg-Marquardt complexity for solving prohibitive for large problems (100 views 10,000 points ~30,000 unknowns) Partition parameters partition A partition B (only dependent on A and itself) typically A contains camera parameters, and B contains 3D points

Sparse bundle adjustment residuals: normal equations: with

Sparse bundle adjustment normal equations: modified normal equations: solve in two parts:

Sparse bundle adjustment U1U1 U2U2 U3U3 WTWT W V P1P1 P2P2 P3P3 M Jacobian of has sparse block structure 12xm 3xn (in general much larger) im.pts. view 1 Needed for non-linear minimization

Sparse bundle adjustment Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m WTWT V U-WV -1 W T 11xm 3xn Allows much more efficient computations e.g. 100 views,10000 points, solve  1000x1000, not  30000x30000 Often still band diagonal use sparse linear algebra algorithms

Sparse bundle adjustment normal equations: modified normal equations: solve in two parts:

Sparse bundle adjustment Covariance estimation

Degenerate configurations Camera resectioning Two views More views (H&Z Ch.21)

Camera resectioning Cameras as points 2D case – Chasles’ theorem

Ambiguity for 3D cameras Twisted cubic (or less) meeting lin. subspace(s) (degree+dimension<3)

Ambiguous two-view reconstructions Ruled quadric containing both scene points and camera centers  alternative reconstructions exist for which the reconstruction of points located off the quadric are not projectively equivalent hyperboloid 1s cone pair of planes single plane + 2 points single line + 2 points

Multiple view reconstructions Single plane is still a problem Hartley and others looked at 3 and more view critical configurations, but those are rather exotic and are not a problem in practice.

Next class: Dynamic structure from motion