Motion Segmentation with Missing Data using PowerFactorization & GPCA Part I 1 Title 2 Motivation 3 Problem statement 4 Brief review of PCA 5 Extensions: PPA, NLPCA, ICA 6 Problem Statement and Prior Work 7 Our approach Part II 8 Simple example 1 9 Simple example 2 10 Notation and embedding 11 Solution part1 12 Solution part2 13 Solution part3 14 Algorithm summary 15 Nonlinear case Part III 16 Simulation results 17 2D Motion Segmentation 18 3D Motion Segmentation 19 Conclusions René Vidal1,2 Richard Hartley2,3 1Biomedical Engineering Johns Hopkins University 2National ICT Australia 3Australian National University

3-D motion segmentation problem Given a set of point correspondences in multiple views, determine Number of motion models Motion model: affine Segmentation: model to which each pixel belongs Mathematics of the problem depends on Number of frames (2, 3, multiple) Projection model (affine, perspective) Motion model (affine, translational, planar motion, rigid motion) 3-D structure (planar or not)

Prior work on 3-D motion segmentation Perspective cameras (statistical approaches) Motion segmentation & model selection (Torr’98) Multiple rigid motions using NCuts+EM (Feng-Perona ’98) Perspective cameras (geometric approaches) Points in a line (Shashua-Levin ‘01) Points in a plane moving linearly at constant speed Points in moving in planes (Sturm ’02) Segmentation of two rigid motions (Wolf-Shashua ‘01) Perspective cameras, multiple rigid-body motions Multibody fundamental matrix (Vidal et al. ECCV’02, CVPR’03) All 2-view motion models (Vidal-Ma ECCV’04) Affine, translational, homographies, fundamental matrices Multibody trifocal tensor (Hartley-Vidal CVPR’04) Multiple views: only for affine cameras Complete references

Single-body factorization Structure = 3D surface Motion = camera position and orientation Affine camera model p = point f = frame Motion of one rigid-body lives in a 4-D subspace (Boult and Brown ’91, Tomasi and Kanade ‘92) P = #points F = #frames

Multi-body factorization Given n rigid motions Motion segmentation is obtained from Leading singular vector of (Boult and Brown ’91) Shape interaction matrix (Costeira & Kanade ’95, Gear ’94) Number of motions (if fully-dimensional) Motion subspaces need to be independent (Kanatani ’01)

Multi-body factorization Sensitive to noise Kanatani (ICCV ’01): use model selection to scale entries of Q Wu et al. (CVPR’01): project data onto subspaces and iterate Fails with partially dependent motions Zelnik-Manor and Irani (CVPR’03) Build similarity matrix from normalized Q Apply spectral clustering to similarity matrix Kanatani (ECCV’04) Assume degeneracy is known: pure translation in the image Segment data by multi-stage optimization (multiple EM problems) Cannot handle missing data Gruber and Weiss (CVPR’04) Expectation Maximization

Our approach: PowerFactorization+GPCA New motion segmentation algorithm that Is provably correct with perfect data Handles both independent and degenerate motions Handles both complete and incomplete data Project trajectories onto a 5-D subspace of Complete data: PCA or SVD Incomplete data: PowerFactorization Cluster projected subspaces using GPCA Non-iterative: can be used to initialize EM

Projection onto a 5-D subspace Motion of one rigid-body lives in 4-D subspace of By projecting onto a 5-D subspace of Number and dimensions of subspaces are preserved Motion segmentation is equivalent to clustering subspaces of dimension 2, 3 or 4 in Minimum #frames = 3 (CK needs a minimum of 2n frames for n motions) Can remove outliers by robustly fitting the 5-D subspace (complete data) What projection to use? 5 principal components Motion 1 Motion 2

Projection onto a 5-D subspace PowerFactorization algorithm: Complete data Given A solve for B Orthonormalize B Given B solve for A Iterate Converges to rank-r approximation with rate Given , factor it as Incomplete data It diverges in some cases Works well with up to 30% of missing data Linear problem

Motion segmentation using GPCA Apply polynomial embedding to 5-D points One can linearly fit the polynomial Veronese map

Motion segmentation using GPCA Theorem: subspace clustering using GPCA Estimate multibody motion model: fit polynomial Estimate motion models: differentiate the polynomial

Motion segmentation using GPCA With a moderate level of noise Polynomials may not be a perfect union of subspaces Normals can estimated correctly by choosing points optimally Distance to closest subspace without knowing segmentation?

Combining GPCA & Spectral Clustering Spectral clustering can be applied to segment Independent motions: equivalent to Costeira-Kanade (Weiss’98) Dependent motions: heuristic similarity (Zelnik-Manor ea ‘03) How can we apply spectral clustering to both independent and degenerate motions? Need measure of similarity: Euclidean distance does not work Apply GPCA to each point to obtain its normal vector Use the angle between normals to obtain similarity

Algorithm summary Projection Motion Segmentation Project point trajectories onto a 5-D subspace using either PCA or PowerFactorization Motion Segmentation Fit a set of homogeneous polynomials representing all motion subspaces to projected trajectories Obtain a basis for each motion subspace from the derivatives of the polynomials Cluster the point trajectories by either Assigning each trajectory to its closest subspace Applying spectral clustering to similarity matrix built from the subspace angles

Experimental results Sequence A Sequence B Sequence C Percentage of correct classification

Experimental results Also tested algorithm on Sequences with up to 30% missing data Sequences with full and degenerate motions Full motions Linear motions Planar motions Sequences with perspective effects Sequences with transparent motions Sequences with minimum #frames

Experimental results

Conclusions PowerFactorization+GPCA: algebraic approach to motion segmentation that Is provably correct with perfect data Handles independent & degenerate motions Handles both complete and incomplete data Gives about 5% misclassification error with up to 30% missing data Faster than multi-stage optimization by EM Open issues Convergence of PF with missing data Dealing with outliers in PF and GPCA Dealing with variable number of motions

A sequence with variable # of motions

Minimum number of correspondences