1. Motion Segmentation with Missing Data using PowerFactorization & GPCA
Part I
1 Title
2 Motivation
3 Problem statement
4 Brief review of PCA
5 Extensions: PPA, NLPCA, ICA
6 Problem Statement and Prior Work
7 Our approach
Part II
8 Simple example 1
9 Simple example 2
10 Notation and embedding
11 Solution part1
12 Solution part2
13 Solution part3
14 Algorithm summary
15 Nonlinear case
Part III
16 Simulation results
17 2D Motion Segmentation
18 3D Motion Segmentation
19 Conclusions
René Vidal1,2
Richard Hartley2,3
1Biomedical Engineering Johns Hopkins University
2National ICT Australia
3Australian National University

2. 3-D motion segmentation problem
Given a set of point correspondences in multiple views, determine
Number of motion models
Motion model: affine
Segmentation: model to which each pixel belongs
Mathematics of the problem depends on
Number of frames (2, 3, multiple)
Projection model (affine, perspective)
Motion model (affine, translational, planar motion, rigid motion)
3-D structure (planar or not)

3. Prior work on 3-D motion segmentation
Perspective cameras (statistical approaches)
Motion segmentation & model selection (Torr’98)
Multiple rigid motions using NCuts+EM (Feng-Perona ’98)
Perspective cameras (geometric approaches)
Points in a line (Shashua-Levin ‘01)
Points in a plane moving linearly at constant speed
Points in moving in planes (Sturm ’02)
Segmentation of two rigid motions (Wolf-Shashua ‘01)
Perspective cameras, multiple rigid-body motions
Multibody fundamental matrix (Vidal et al. ECCV’02, CVPR’03)
All 2-view motion models (Vidal-Ma ECCV’04)
Affine, translational, homographies, fundamental matrices
Multibody trifocal tensor (Hartley-Vidal CVPR’04)
Multiple views: only for affine cameras
Complete references

4. Single-body factorization
Structure = 3D surface
Motion = camera position and orientation
Affine camera model
p = point
f = frame
Motion of one rigid-body lives in a 4-D subspace (Boult and Brown ’91, Tomasi and Kanade ‘92)
P = #points
F = #frames

5. Multi-body factorization
Given n rigid motions
Motion segmentation is obtained from
Leading singular vector of (Boult and Brown ’91)
Shape interaction matrix (Costeira & Kanade ’95, Gear ’94)
Number of motions (if fully-dimensional)
Motion subspaces need to be independent (Kanatani ’01)

6. Multi-body factorization
Sensitive to noise
Kanatani (ICCV ’01): use model selection to scale entries of Q
Wu et al. (CVPR’01): project data onto subspaces and iterate
Fails with partially dependent motions
Zelnik-Manor and Irani (CVPR’03)
Build similarity matrix from normalized Q
Apply spectral clustering to similarity matrix
Kanatani (ECCV’04)
Assume degeneracy is known: pure translation in the image
Segment data by multi-stage optimization (multiple EM problems)
Cannot handle missing data
Gruber and Weiss (CVPR’04)
Expectation Maximization

7. Our approach: PowerFactorization+GPCA
New motion segmentation algorithm that
Is provably correct with perfect data
Handles both independent and degenerate motions
Handles both complete and incomplete data
Project trajectories onto a 5-D subspace of
Complete data: PCA or SVD
Incomplete data: PowerFactorization
Cluster projected subspaces using GPCA
Non-iterative: can be used to initialize EM

8. Projection onto a 5-D subspace
Motion of one rigid-body lives in 4-D subspace of
By projecting onto a 5-D subspace of
Number and dimensions of subspaces are preserved
Motion segmentation is equivalent to clustering subspaces of dimension 2, 3 or 4 in
Minimum #frames = 3 (CK needs a minimum of 2n frames for n motions)
Can remove outliers by robustly fitting the 5-D subspace (complete data)
What projection to use?
5 principal components
Motion 1
Motion 2

9. Projection onto a 5-D subspace
PowerFactorization algorithm:
Complete data
Given A solve for B
Orthonormalize B
Given B solve for A
Iterate
Converges to rank-r approximation with rate
Given , factor it as
Incomplete data
It diverges in some cases
Works well with up to 30% of missing data
Linear problem

10. Motion segmentation using GPCA
Apply polynomial embedding to 5-D points
One can linearly fit the polynomial
Veronese map

11. Motion segmentation using GPCA
Theorem: subspace clustering using GPCA
Estimate multibody motion model: fit polynomial
Estimate motion models: differentiate the polynomial

12. Motion segmentation using GPCA
With a moderate level of noise
Polynomials may not be a perfect union of subspaces
Normals can estimated correctly by choosing points optimally
Distance to closest subspace without knowing segmentation?

13. Combining GPCA & Spectral Clustering
Spectral clustering can be applied to segment
Independent motions: equivalent to Costeira-Kanade (Weiss’98)
Dependent motions: heuristic similarity (Zelnik-Manor ea ‘03)
How can we apply spectral clustering to both independent and degenerate motions?
Need measure of similarity: Euclidean distance does not work
Apply GPCA to each point to obtain its normal vector
Use the angle between normals to obtain similarity

14. Algorithm summary Projection Motion Segmentation
Project point trajectories onto a 5-D subspace using either PCA or PowerFactorization
Motion Segmentation
Fit a set of homogeneous polynomials representing all motion subspaces to projected trajectories
Obtain a basis for each motion subspace from the derivatives of the polynomials
Cluster the point trajectories by either
Assigning each trajectory to its closest subspace
Applying spectral clustering to similarity matrix built from the subspace angles

15. Experimental results Sequence A Sequence B Sequence C
Percentage of correct classification

16. Experimental results Also tested algorithm on
Sequences with up to 30% missing data
Sequences with full and degenerate motions
Full motions
Linear motions
Planar motions
Sequences with perspective effects
Sequences with transparent motions
Sequences with minimum #frames

17. Experimental results

18. Conclusions
PowerFactorization+GPCA: algebraic approach to motion segmentation that
Is provably correct with perfect data
Handles independent & degenerate motions
Handles both complete and incomplete data
Gives about 5% misclassification error with up to 30% missing data
Faster than multi-stage optimization by EM
Open issues
Convergence of PF with missing data
Dealing with outliers in PF and GPCA
Dealing with variable number of motions

19. A sequence with variable # of motions

20. Minimum number of correspondences