Two-View Motion Segmentation by Exploiting the Rank and Geometry of the Multibody Fundamental Matrix René Vidal and Xiaodong Fan Center for Imaging Science Johns Hopkins University Add d15.avi

Structure and motion recovery problem Input: Corresponding points in multiple images Output: camera motion, scene structure, camera calibration Structure = 3D surface Motion = camera position and orientation Copyright @ Rene Vidal

The fundamental matrix Input: point correspondences Output: rigid motion Rotation: Translation: Epipolar constraint Estimate fundamental matrix F linearly Eight-point algorithm Project onto the essential manifold for noise reduction Recover (R,T) from F Write Sym(F_1,…., F_n). Copyright @ Rene Vidal

Projection onto the essential manifold Characterizing essential manifold F Extend to multiple motions! Projection – minimizing the Frobenious norm: Copyright @ Rene Vidal

3-D motion segmentation problem Given a set of point correspondences in multiple views, determine Number of motion models Motion model: affine Segmentation: model to which each pixel belongs Mathematics of the problem depends on Number of frames (2, 3, multiple) Projection model (affine, perspective) Motion model (affine, translational, planar motion, rigid motion) 3-D structure (planar or not) Copyright @ Rene Vidal

Our approach to motion segmentation Estimation of multiple motion models equivalent to estimation of one multibody motion model Eliminate feature clustering: multiplication Estimate a single multibody motion model: polynomial fitting Segment multibody motion model: polynomial differentiation chicken-and-egg Copyright @ Rene Vidal

A unified approach to motion segmentation Applies to most motion models in computer vision All motion models can be segmented algebraically by Fitting multibody model: real or complex polynomial to all data Fitting individual model: differentiate polynomial at a data point Copyright @ Rene Vidal

Multibody epipolar constraint Rotation: Translation: Epipolar constraint Multiple motions Multibody epipolar constraint Satisfied by ALL points regardless of segmentation Segmentation is algebraically eliminated!!! Copyright @ Rene Vidal

Multibody fundamental matrix Lifting Embedding Bilinear on embedded data! Veronese map (polynomial embedding) Multibody fundamental matrix Copyright @ Rene Vidal

Multibody Fundamental Matrix n-body motion 1-body motion Copyright @ Rene Vidal

Epipoles in the nullspace of F Embedded epipoles are in the null space of the multibody fundamental matrix Are all vectors in Null(F) embedded epipoles, or their linear combinations? Are embedded epipoles linearly independent? F The dimension of Null(F) Copyright @ Rene Vidal

Rank constraint of F 1-body motion 2-body motion In general, we prove: # of distinct epipoles Key observation: repeated epipoles will enlarge the null(F) ! time each eipipole repeates Copyright @ Rene Vidal

Dimension of Null(F) Let be Different epipoles (up to scales), each of which repeats times Define as the span of the l-th order derivatives of Copyright @ Rene Vidal

Dimension of Null(F) Lemma 1: Lemma 2: when Lemma 3: Copyright @ Rene Vidal

Projection onto the essential manifold Characterizing essential manifold F Extend to multiple motions! Projection – minimizing the Frobenious norm: Copyright @ Rene Vidal

Multibody essential manifold with common rotations n-body motion 1-body motion Copyright @ Rene Vidal

SVD of When the number of motions is odd When there are two motions Theorem: The singular values of a skew-symmetric matrix are When there are two motions Theorem: The singular values satisfies Copyright @ Rene Vidal

Projection onto the multibody essential manifold Based on the proposed projection theorem, the projection process is: Linearly estimate F Compute SVD Replace by the “desired” singular values Satisfy the multi-body essential manifold constraint Copyright @ Rene Vidal

Compute the “desired” singular values Using Lagrangian multiplier method to solve the constraint optimization problem Two motions with a common rotation Odd number of motions Copyright @ Rene Vidal

Multibody 8-point algorithm Multibody epipolar constraint Multibody epipolar transfer Multibody epipole Fundamental matrices Given rank condition n linear system F Lifting Embedding Sym(F1,,,F_n) This can be solved using a collection of GPCA’s Copyright @ Rene Vidal

Experiments Compare three methods: GPCA (Vidal and Ma, 2004): specifically designed for the purely translational motions. Multibody epipolar constraint without projection (MEC-noprojection): using the proposed approach without projecting the estimated multibody fundamental matrix F onto the multibody essential manifold. Multibody epipolar constraint with projection (MEC-projection): project the estimated multibody fundamental matrix F onto the multibody essential manifold. Copyright @ Rene Vidal

Experimental results on synthesized data Error in the estimation of translational component of motion (in degree) as a function of noise in the image points (std in pixels) for two pure translational motions (left column); and as a function of rotation angle (in degree) for two independent motions with a common rotation (right column). (a) in translation Copyright @ Rene Vidal

Experimental results on synthesized data Error in the estimation of rotational component of motion (in degree) as a function of noise in the image points (std in pixels) for two pure translational motions (left column); and as a function of rotation angle (in degree) for two independent motions with a common rotation (right column). (b) in rotation Copyright @ Rene Vidal

Experimental results on synthesized data Error in the segmentation (in percentage) as a function of noise in the image points (std in pixels) for two pure translational motions (left column); and as a function of rotation angle (in degree) for two independent motions with a common rotation (right column). (c) in segmentation Copyright @ Rene Vidal

Experimental results on real images Experiment setup: The original sequence contains three independent motions, the first two of which are translational. Point correspondences, Compute the rotational component R of the third motion. Rotate the feature points in the first frame that belong to the first motion, Then undergo a two-body motion with a common rotation. Copyright @ Rene Vidal

Experimental results on real images Percentage of mis-segmentation on a real sequence with two independent motions (with a common rotation) for different pairs of frames. The x-axis indicates the index of frame pairs. Copyright @ Rene Vidal

Conclusions Unified algebraic approach to motion segmentation Fit a polynomial to all image data Differentiate the polynomial to obtain motion parameters Applies to most motion models in vision Two views 2-D: translational, similarity, affine 3-D: translational, fundamental matrices, homographies Three views Multibody trifocal tensor Multiple views Affine cameras Two perspective views Rank of multibody fundamental matrices Geometry of multibody fundamental matrices Copyright @ Rene Vidal