Projective structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, …

Last class: Affine camera The affine projection equations are how to find the origin? or for that matter a 3D reference point? affine projection preserves center of gravity

Orthographic factorization The orthographic projection equations are where All equations can be collected for all i and j where Note that P and X are resp. 2mx3 and 3xn matrices and therefore the rank of x is at most 3

Orthographic factorization Factorize m through singular value decomposition An affine reconstruction is obtained as follows Closest rank-3 approximation yields MLE!

Tentative class schedule Jan 16/18 - Introduction Jan 23/25 Cameras Radiometry Jan 30/Feb1 Sources & Shadows Color Feb 6/8 Linear filters & edges Texture Feb 13/15 Multi-View Geometry Stereo Feb 20/22 Optical flow Project proposals Feb27/Mar1 Affine SfM Projective SfM Mar 6/8 Camera Calibration Silhouettes and Photoconsistency Mar 13/15 Springbreak Mar 20/22 Segmentation Fitting Mar 27/29 Prob. Segmentation Project Update Apr 3/5 Tracking Apr 10/12 Object Recognition Apr 17/19 Range data Apr 24/26 Final project

PROJECTIVE STRUCTURE FROM MOTION The Projective Structure from Motion Problem Elements of Projective Geometry Projective Structure and Motion from Two Images Projective Motion from Fundamental Matrices Projective Structure and Motion from Multiple Images Reading: Chapter 13.

The Projective Structure-from-Motion Problem Given m perspective images of n fixed points P we can write j Problem: estimate the m 3x4 matrices M and the n positions P from the mn correspondences p . i j ij 2mn equations in 11m+3n unknowns Overconstrained problem, that can be solved using (non-linear) least squares!

The Projective Ambiguity of Projective SFM When the intrinsic and extrinsic parameters are unknown If M and P are solutions, i j So are M’ and P’ where i j and Q is an arbitrary non-singular 4x4 matrix. Q is a projective transformation.

Projective Spaces: (Semi-Formal) Definition

A Model of P( R ) 3

Projective Subspaces and Projective Coordinates

Projective Subspaces and Projective Coordinates

Projective Subspaces Given a choice of coordinate frame Line: Plane:

Affine and Projective Spaces

Affine and Projective Coordinates

Cross-Ratios Collinear points Pencil of coplanar lines Pencil of planes {A,B;C,D}= sin(+)sin(+) sin(++)sin

Cross-Ratios and Projective Coordinates * Along a line equipped with the basis * In a plane equipped with the basis In 3-space equipped with the basis

Projective Transformations Bijective linear map: Projective transformation: ( = homography ) Projective transformations map projective subspaces onto projective subspaces and preserve projective coordinates. Projective transformations map lines onto lines and preserve cross-ratios.

Perspective Projections induce projective transformations between planes.

Geometric Scene Reconstruction Idea: use (A,O”,O’,B,C) as a projective basis.

Reprinted from “Relative Stereo and Motion Reconstruction,” by J Reprinted from “Relative Stereo and Motion Reconstruction,” by J. Ponce, T.A. Cass and D.H. Marimont, Tech. Report UIUC-BI-AI-RCV-93-07, Beckman Institute, Univ. of Illinois (1993).

Motion estimation from fundamental matrices Q Once M and M’ are known, P can be computed with LLS. Facts: b’ can be found using LLS.

Projective Structure from Motion and Factorization Algorithm (Sturm and Triggs, 1996) Guess the depths; Factorize D; Iterate. Does it converge? (Mahamud and Hebert, 2000)

Bundle adjustment - refining structure and motion Minimize reprojection error Maximum Likelyhood Estimation (if error zero-mean Gaussian noise) Huge problem but can be solved efficiently (exploit sparseness)

Bundle adjustment Developed in photogrammetry in 50´s

Non-linear least squares Linear approximation of residual allows quadratic approximation of sum- of-squares Minimization corresponds to finding zeros of derivative N Levenberg-Marquardt: extra term to deal with singular N (decrease/increase l if success/failure to descent) (extra term = descent term)

Bundle adjustment Jacobian of has sparse block structure cameras independent of other cameras, points independent of other points U1 U2 U3 WT W V P1 P2 P3 M im.pts. view 1 12xm 3xn (in general much larger) Needed for non-linear minimization

Bundle adjustment Allows much more efficient computations Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m WT V U-WV-1WT 11xm 3xn Allows much more efficient computations e.g. 100 views,10000 points, solve 1000x1000, not 30000x30000 Often still band diagonal use sparse linear algebra algorithms

Sequential SfM Initialize motion from two images Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

Initial projective camera motion Choose P and P´compatible with F Reconstruction up to projective ambiguity (reference plane;arbitrary) Same for more views? Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion different projective basis

Initializing projective structure Reconstruct matches in projective frame by minimizing the reprojection error Non-iterative optimal solution Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

Projective pose estimation Infere 2D-3D matches from 2D-2D matches Compute pose from (RANSAC,6pts) X F x Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion Inliers:

Refining and extending structure Refining structure Extending structure 2-view triangulation (Iterative linear) Initialize motion Initialize structure For each additional view Determine pose of camera Refine and extend structure Refine structure and motion

Refining structure and motion use bundle adjustment Also model radial distortion to avoid bias!

Hierarchical structure and motion recovery Compute 2-view Compute 3-view Stitch 3-view reconstructions Merge and refine reconstruction F T H PM

Metric structure and motion use self-calibration (see next class) Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views) (solution possible based on model selection)

Dealing with dominant planes

PPPgric HHgric

Farmhouse 3D models (note: reconstruction much larger than camera field-of-view)

Application: video augmentation

Next class: Camera calibration (and self-calibration) Reading: Chapter 2 and 3