1. A Unified Algebraic Approach to 2D and 3D Motion Segmentation
René Vidal
Center for Imaging Science
Johns Hopkins University

2. Motion segmentation: 2 views
A static scene: multiple 2D motion models
A dynamic scene: multiple 3D motion models
Given an image sequence, determine
Number of motion models (affine, Euclidean, etc.)
Motion model: affine (2D) or Euclidean (3D)
Segmentation: model to which each pixel belongs

3. Computer Vision Structure from motion and 3D reconstruction
Input: Corresponding points in multiple images
Output: camera motion, Euclidean scene structure
Theory
Multiview geometry: Multiple view matrix
Multiple view normalized epipolar constraint
Linear self-calibration
Algorithms
Multiple view matrix factorization algorithm
Multiple view factorization for planar motions

4. Multibody Multiview Geometry
Given an image sequence, determine
Number of motion models (affine, Euclidean, etc.)
Motion estimation: affine (2D) or Euclidean (3D)
Data segmentation: model associated with each pixel
Prior work 3D multibody multiple view geometry
Points in a line
Points in a conic
Coplanar points linearly moving at constant speed
Points in multiple planes
Multibody Structure from Motion
3D Motion Segmentation: multibody epipolar constraint
Affine Motion Segmentation: multibody brightness constancy and affine constraints

5. Multibody Multiview Geometry
Multibody Structure from Motion
3D Motion Segmentation: multibody epipolar constraint
Affine Motion Segmentation: multibody brightness constancy and affine constraints
Generalized PCA
Segmentation of mixtures of subspaces
Image/video segmentation

6. Motion-based image segmentation
Two motions
Camera panning to the right
Car translating to the right

7. Segmentation of linear motions
Multiple objects translating in 3D

8. Piecewise Bilinear Data Multibody structure from motion

9. Motion Segmentation: bilinear data
Rotation:
Translation:
Epipolar constraint
Multiple motions
Write Sym(F_1,…., F_n).
Multibody epipolar constraint

10. Estimation of fundamental matrices
Multibody epipolar constraint
Lifting
Embedding
Given
rank condition n
linear system F
Theorem: Multibody structure from motion [Vidal et al.]
Factorization of bilinear forms can be reduced to factorization of linear forms
Estimation of fundamental matrices can be reduced to GPCA

11. 3D Motions: Multibody epipolar transfer
Number of motions
Lifting
Multibody epipolar line
Polynomial factorization

12. 3D Motions: Multibody epipole
Lifting
The multibody epipole is the solution of the linear system
Number of distinct epipoles
Epipoles are obtained using polynomial factorization

13. 3D Motions: Fundamental matrices
Columns of are epipolar lines
Polynomial factorization to compute them up to scale
Scales can be computed linearly

14. Optimal 3D motion segmentation
Zero-mean Gaussian noise
Constrained optimization problem on
Optimal function for 1 motion
Optimal function for n motions
Solved using Riemanian Gradient Descent

15. Comparison of 1 and n bodies

16. 3D Motion Segmentation Results

17. Affine Motion Segmentation Results

18. Conclusions
There is an algebraic/geometric solution to simultaneous model estimation and data segmentation for
Mixtures of subspaces: linear constraints
Motion segmentation: bilinear constraints
Solution based on
Polynomial factorization: linear algebra
Solution is closed form if ngroups ≤ 4
Showed applications in
Image segmentation: intensity and texture
Video segmentation: affine and 3D motion segmentation

19. Ongoing work and future directions
Machine Learning and Statistical Geometry
Robust GPCA
Connections with learning methods: Kernel PCA, etc.
Model selection: different classes of models
Estimating manifolds from sample data points
Applications of GPCA in Computer Vision
Cue integration
Multiple view geometry of dynamic scenes
Shape recognition: faces
A geometric/statistical theory of segmentation?

20. Dynamic GPCA: Recognition/Synthesis of Human Motion Recognition/Synthesis of Dynamic Textures
Given image data
Estimate a mixture of linear dynamical models (linear hybrid systems)
Use the models to segment/recognize
Human activity
Dynamic texture
Use the models to synthesize
human motion
dynamic textures

21. Thanks Computer Vision Pursuit-Evasion Games Vision Based Landing
Stefano Soatto, UCLA
Yi Ma, UIUC
Jana Kosecka, GMU
John Oliensis, NEC
Control/Hybrid Systems
John Lygeros, Cambridge
Shawn Schaffert
Research Advisor
Shankar Sastry
Pursuit-Evasion Games
Jin Kim
David Shim
Vision Based Landing
Omid Shakernia
Cory Sharp
Formation Control
Noah Cowan

22. Multiple View Geometry (MVG)
Obtain camera motion and scene structure from multiple images of a cloud of 3D feature points

23. MVG: Anatomy of cases (state of the art)
surface
curve
line
point
theory
algorithm
practice
Euclidean
affine
projective
2 views
3 views
4 views
m views
algebra
geometry
optimization

24. MVG: A need for unification
Euclidean
surface
curve
line
point
2 views
3 views
4 views
m views
theory
algorithm
practice
affine
projective
algebra
geometry
optimization
rank deficiency of
Multiple View Matrix

25. MVG: The Multiple View Matrix
Relationship between first and i-th views
Theorem: [Rank deficiency of Multiple View Matrix]
Theorem: [Dependency of multilinear constraints]
Constraints among more than three views are algebraically dependent (quadrilinear in particular)
(degenerate)
(generic)

26. Texture segmentation Given a static image, determine Number of groups
Segmentation: pixels that have the same texture

27. Generalized Principal Component Analysis (GPCA)
Given data lying on a collection of subspaces
Number of subspaces
Model for each subspace: basis
Segmentation: model to which each point belongs

28. Vision Based Formation Control
Green follows red
Blue follows green