Computer Vision Structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, A. Zisserman, R. Hartley, Luc Van Gool, …

Computer Vision Last time: Optical Flow u IxIx ItIt IxuIxu I x u=  I t Aperture problem: two solutions: - regularize (smoothness prior) - constant over window (i.e. Lucas-Kanade) Coarse-to-fine, parametric models, etc…

Computer Vision Aug 26/28-Introduction Sep 2/4CamerasRadiometry Sep 9/11Sources & ShadowsColor Sep 16/18Linear filters & edges(Isabel hurricane) Sep 23/25Pyramids & TextureMulti-View Geometry Sep30/Oct2StereoProject proposals Oct 7/9Tracking (Welch)Optical flow Oct 14/16-- Oct 21/23Silhouettes/carving(Fall break) Oct 28/30-Structure from motion Nov 4/6Project updateCamera calibration Nov 11/13SegmentationFitting Nov 18/20Prob. segm.&fit.Matching templates Nov 25/27Matching relations(Thanksgiving) Dec 2/4Range dataFinal project Tentative class schedule

Computer Vision Today’s menu Affine structure from motion –Geometric construction –Factorization Projective structure from motion –Factorization –Sequential

Computer Vision Affine Structure from Motion Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8: (1990).  1990 Optical Society of America. Given m pictures of n points, can we recover the three-dimensional configuration of these points? the camera configurations? (structure) (motion)

Computer Vision Orthographic Projection Parallel Projection

Computer Vision Weak-Perspective Projection Paraperspective Projection

Computer Vision The Affine Structure-from-Motion Problem Given m images of n fixed points P we can write Problem: estimate the m 2x4 matrices M and the n positions P from the mn correspondences p. i j ij 2mn equations in 8m+3n unknowns Overconstrained problem, that can be solved using (non-linear) least squares! j

Computer Vision The Affine Ambiguity of Affine SFM If M and P are solutions, i j So are M’ and P’ where i j and Q is an affine transformation. When the intrinsic and extrinsic parameters are unknown

Computer Vision Affine Spaces: (Semi-Formal) Definition

Computer Vision Example: R as an Affine Space 2

Computer Vision In General The notation is justified by the fact that choosing some origin O in X allows us to identify the point P with the vector OP. Warning: P+u and Q-P are defined independently of O!!

Computer Vision Barycentric Combinations Can we add points? R=P+Q NO! But, when we can define Note:

Computer Vision Affine Subspaces

Computer Vision Affine Coordinates Coordinate system for U: Coordinate system for Y=O+U: Coordinate system for Y: Affine coordinates: Barycentric coordinates:

Computer Vision When do m+1 points define a p-dimensional subspace Y of an n-dimensional affine space X equipped with some coordinate frame basis? Writing that all minors of size (p+2)x(p+2) of D are equal to zero gives the equations of Y. Rank ( D ) = p+1, where

Computer Vision Affine Transformations Bijections from X to Y that: map m-dimensional subspaces of X onto m-dimensional subspaces of Y; map parallel subspaces onto parallel subspaces; and preserve affine (or barycentric) coordinates. In E they are combinations of rigid transformations, non- uniform scalings and shears. Bijections from X to Y that: map lines of X onto lines of Y; and preserve the ratios of signed lengths of line segments. 3

Computer Vision Affine Transformations II Given two affine spaces X and Y of dimension m, and two coordinate frames (A) and (B) for these spaces, there exists a unique affine transformation mapping (A) onto (B). Given an affine transformation from X to Y, one can always write: When coordinate frames have been chosen for X and Y, this translates into:

Computer Vision Affine projections induce affine transformations from planes onto their images.

Computer Vision Affine Shape Two point sets S and S’ in some affine space X are affinely equivalent when there exists an affine transformation  : X X such that X’ =  ( X ). Affine structure from motion = affine shape recovery. = recovery of the corresponding motion equivalence classes.

Computer Vision Geometric affine scene reconstruction from two images (Koenderink and Van Doorn, 1991).

Computer Vision Affine Structure from Motion (Koenderink and Van Doorn, 1991) Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8: (1990).  1990 Optical Society of America.

Computer Vision The Affine Epipolar Constraint Note: the epipolar lines are parallel.

Computer Vision Affine Epipolar Geometry

Computer Vision The Affine Fundamental Matrix where

Computer Vision An Affine Trick.. Algebraic Scene Reconstruction

Computer Vision The Affine Structure of Affine Images Suppose we observe a scene with m fixed cameras.. The set of all images of a fixed scene is a 3D affine space!

Computer Vision has rank 4!

Computer Vision From Affine to Vectorial Structure Idea: pick one of the points (or their center of mass) as the origin.

Computer Vision What if we could factorize D? (Tomasi and Kanade, 1992) Affine SFM is solved! Singular Value Decomposition We can take

Computer Vision From uncalibrated to calibrated cameras Weak-perspective camera: Calibrated camera: Problem: what is Q ? Note: Absolute scale cannot be recovered. The Euclidean shape (defined up to an arbitrary similitude) is recovered.

Computer Vision Reconstruction Results (Tomasi and Kanade, 1992) Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi and T. Kanade, Proc. IEEE Workshop on Visual Motion (1991).  1991 IEEE.

Computer Vision More examples Tomasi Kanade’92, Poelman & Kanade’94

Computer Vision More examples Tomasi Kanade’92, Poelman & Kanade’94

Computer Vision More examples Tomasi Kanade’92, Poelman & Kanade’94

Computer Vision Further Factorization work Factorization with uncertainty Factorization for indep. moving objects Factorization for dynamic objects Perspective factorization (next week) Factorization with outliers and missing pts. (Irani & Anandan, IJCV’02) (Costeira and Kanade ‘94) (Bregler et al. 2000, Brand 2001) (Jacobs 1997 (affine), Martinek and Pajdla 2001, Aanaes 2002 (perspective)) (Sturm & Triggs 1996, …)

Computer Vision Multiple indep. moving objects

Computer Vision Multiple indep. moving objects

Computer Vision Dynamic structure from motion Extend factorization approaches to deal with dynamic shapes (Bregler et al ’00; Brand ‘01)

Computer Vision Representing dynamic shapes represent dynamic shape as varying linear combination of basis shapes (fig. M.Brand)

Computer Vision Projecting dynamic shapes (figs. M.Brand) Rewrite:

Computer Vision Dynamic image sequences One image: Multiple images (figs. M.Brand)

Computer Vision Dynamic SfM factorization? Problem: find J so that M has proper structure

Computer Vision Dynamic SfM factorization (Bregler et al ’00) Assumption: SVD preserves order and orientation of basis shape components

Computer Vision Results (Bregler et al ’00)

Computer Vision Dynamic SfM factorization (Brand ’01) constraints to be satisfied for M constraints to be satisfied for M, use to compute J hard! (different methods are possible, not so simple and also not optimal)

Computer Vision Non-rigid 3D subspace flow Same is also possible using optical flow in stead of features, also takes uncertainty into account (Brand ’01)

Computer Vision Results (Brand ’01)

Computer Vision (Brand ’01) Results

Computer Vision Results (Bregler et al ’01)

Computer Vision Next class: Projective structure from motion