1. Segmentation of Dynamic Scenes
René Vidal
Dept. of EECS, UC Berkeley
Yi Ma (UIUC) Stefano Soatto (UCLA) Shankar Sastry (UCB)

2. Outline Data segmentation: linear case
One dimension: eigenvector segmentation
K-dimensions: Generalized PCA
Motion segmentation: bilinear case
3D Motion Segmentation
Affine Motion Segmentation
Examples and Control Applications

3. Motivation Given a set of image points obtain:
Number of independently moving objects
Segmentation: object to which each point belongs
Motion: rotation and translation of each object
Structure: depth of each point

4. Previous Work 2-D motion segmentation
Look for flow discontinuities
Apply normalized cuts to similarity matrix
Model selection and EM
3-D geometric motion segmentation
Orthographic factorization: Costeira & Kanade ’98
Points in a conic:
Avidan-Shashua ’01
Han and Kanade ’00
Points in a line: Levin-Shashua ‘01
2-body motions: Wolf-Shashua ‘01

5. Our Approach to Segmentation
We propose an algebraic geometric approach to data segmentation
Number of groups = degree of a polynomial
Groups ≈ roots of a polynomial
= polynomial factorization
We show that
There exist segmentation independent constraints
There exists a unique closed form solution if n<5
The exact solution can be computed using linear algebra

6. One-dimensional Segmentation

7. One-dimensional Segmentation
For n groups
Number of groups
Groups

8. One-dimensional Segmentation
Solution is unique if
Solution is closed form if
Solves the eigenvector segmentation problem e.g. normalized cuts

9. K-dimensional Segmentation: Generalized PCA via Polynomial Factorization
Data lies on K-1 dimensional subspaces
Generalized PCA (Vidal-Ma-Sastry ‘02)
Solve for the roots of a polynomial of degree n in one variable
Solve for a linear system in n variables

10. Motion Segmentation: 2 views
Two-view motion constraints are bilinear
Affine motion segmentation:
constant brightness constraint
3D motion segmentation:
epipolar constraint

11. Motion Segmentation: 2 views
Multibody affine and epipolar constraints

12. Multibody Affine and Fundamental Matrix
Lifting
Embedding
Multibody epipolar constraint

13. Multibody Affine and Fundamental Matrix
1-body motion
n-body motion

14. Number of Independent Motions
Theorem: Given image points corresponding to motions, if at least 8 (6) points correspond to each object, then
Minimum number of points
Affine motion 1 2 4 3 6 24 64
160 8 35 99
225
3D motion

15. Multibody epipolar transfer
3D Motion Segmentation
Given
rank condition n
linear system F
Multibody epipolar transfer
Multibody epipole
Fundamental matrices

16. 3D Motions: Multibody epipolar transfer
Lifting
Multibody epipolar line
Polynomial factorization

17. 3D Motions: Multibody epipole
Lifting
The multibody epipole is the solution of the linear system
Number of distinct epipoles
Epipoles are obtained using polynomial factorization

18. 3D Motions: Fundamental matrices
Columns of are epipolar lines
Polynomial factorization to compute them up to scale
Scales can be computed linearly

19. Multibody SFM Algorithm
Image point
Veronese map
Embedded image point
Multibody epipolar transfer
Multibody epipolar line
Polynomial Factorization
Epipolar lines
Linear system
Multibody epipole
Polynomial Factorization
Epipoles
Linear system
Fundamental matrix

20. Affine Motion Segmentation
Given
rank condition n
linear system A
Affine Matrices
2 Rows and 1 Column
Polynomial Factorization

21. Affine Motion Segmentation
Affine motion constraint
Nonlinear algebraic error

22. Example 1: 3D Motion Segmentation

23. Example 2: Affine Motion Segmentation

24. Example 3: Affine Motion Segmentation

25. Example 4: Multiple View Segmentation

26. Experimental Results
Add details from “Experiment section” on Rene’s paper (Tech Report: A Factorization Method for 3D Multi-body Motion Estimation & Segmentation)

27. Experimental Results

28. Example 5: Omnidirectional Segmentation
Shakernia-Vidal-Sastry (Workshop on Motion, 2002)

29. Applications
Vision-based landing (Shakernia-Vidal-Sharp-Ma-Sastry, ICRA 2002)

30. Applications
Pursuit-evasion games (Vidal-Shakernia-Kim- Shim-Sastry, Trans. on Robotics & Automation 2002)

31. Applications
Formation control Vidal-Shakernia-Sastry, 2002

32. Conclusions There is an algebraic/geometric solution to
Data segmentation: linear constraints
Motion segmentation: bilinear constraints
Solution based on
Polynomial factorization: linear algebra
Solution is closed form if n<5
Showed applications in
Pursuit-evasion games & Formation control

33. Linearly moving objects1 2 10 5 20 65 4 3 8 35 99
225
Minimum number of points
Multibody epipole
Recovery of epipoles
Fundamental matrices
Feature segmentation