1. Segmentation of Dynamical Scenes
René Vidal
BioEngineering Department, John Hopkins
Yi Ma
Electrical & Computer Engineering, UIUC

2. One-body two-views

3. One-body multiple-views

4. Motivation and problem statement
A static scene: multiple 2D motion models
A dynamic scene: multiple 3D motion models
Given an image sequence, determine
Number of motion models (affine, Euclidean, etc.)
Motion model: affine (2D) or Euclidean (3D)
Segmentation: model to which each pixel belongs

5. Previous work on 2D motion segmentation
Local methods (Wang-Adelson ’93)
Estimate one model per pixel using data in a window
Cluster models with K-means
Iterate
Aperture problem
Motion across boundaries
Global methods (Irani-Peleg ‘92)
Dominant motion: fit one motion model to all pixels
Look for misaligned pixels & fit a new model to them
Normalized cuts (Shi-Malik ‘98)
Similarity matrix based on motion profile
Segment pixels using eigenvector

6. Previous work on 3D motion segmentation
Factorization techniques
Orthographic/discrete: Costeira-Kanade ’98, Gear ‘98
Perspective/continuous: Vidal-Soatto-Sastry ’02
Omnidirectional/continuous: Shakernia-Vidal-Sastry ’03
Special cases:
Points in a line (orth-discrete): Han and Kanade ’00
Points in a conic (perspective): Avidan-Shashua ’01
Points in a line (persp.-continuous): Levin-Shashua ’01
2-body case: Wolf-Shashua ‘01

7. Previous work: probabilistic techniques
Probabilistic approaches
Generative model: data membership + motion model
Obtain motion models using Expectation Maximization
E-step: Given motion models, segment image data
M-step: Given data segmentation, estimate motion models
2D Motion Segmentation
Layered representation (Jepson-Black’93, Ayer-Sawhney ’95, Darrel-Pentland’95, Weiss-Adelson’96, Weiss’97, Torr-Szeliski-Anandan ’99)
3D Motion Segmentation
EM+Reprojection Error: Feng-Perona’98
EM+Model Selection: Torr ’98
How to initialize iterative algorithms?

8. Our approach to motion segmentation
Image points
Number of motions
Multibody Fund. Matrix
Epipolar lines
Epipoles
Fundamental Matrices
Motion segmentation
Multi epipolar lines
Multi epipole
This work considers
full perspective projection
multiple objects
general motions
We show that
Problem is equivalent to polynomial factorization
There is a unique global closed form solution if n<5
Exact solution is obtained using linear algebra
Can be used to initialize EM-based algorithms

9. One-dimensional segmentation
Number of models?

10. One-dimensional segmentation
For n groups
Number of groups
Groups

11. One-dimensional segmentation
Solution is unique if
Solution is closed form if and only if
Solves the eigenvector segmentation problem e.g. normalized cuts

12. Three-dimensional motion segmentation
Generalized PCA (Vidal, Ma ’02)
Solve for the roots of a polynomial of degree in one variable
Solve for a linear system in variables

13. The multibody epipolar constraint
Rotation:
Translation:
Epipolar constraint
Multiple motions
Multibody epipolar constraint
Satisfied by ALL points regardless of segmentation
Segmentation is algebraically eliminated!!!

14. The multibody fundamental matrix
Lifting
Embedding
Bilinear on embedded data!
Veronese map (polynomial embedding)
Multibody fundamental matrix

15. Estimation of the number of motions
Theorem: Given image points corresponding to motions, if at least 8 points correspond to each object, then 1 2 4 3 8 35 99
225
Minimum number of points

16. Estimation of multibody fundamental matrix
1-body motion
n-body motion

17. Segmentation of fundamental matrices
Given
rank condition for n motions
linear system F
Multibody epipolar transfer
Multibody epipole
Fundamental matrices

18. Multibody epipolar transfer
Lifting
Multibody epipolar line
Polynomial factorization

19. Multibody epipole Lifting
The multibody epipole is the solution of the linear system
Number of distinct epipoles
Epipoles are obtained using polynomial factorization

20. From images to epipoles

21. Fundamental matrices Columns of are epipolar lines
Polynomial factorization to compute them up to scale
Scales can be computed linearly

22. The multibody 8-point algorithm
Image point
Veronese map
Embedded image point
Multibody epipolar transfer
Multibody epipolar line
Polynomial Factorization
Epipolar lines
Linear system
Multibody epipole
Polynomial Factorization
Epipoles
Linear system
Fundamental matrix

23. Optimal 3D motion segmentation
Zero-mean Gaussian noise
Constrained optimization problem on
Optimal function for 1 motion
Optimal function for n motions
Solved using Riemanian Gradient Descent

24. Comparison of 1 body and n bodies

25. Other cases: linearly moving objects1 2 10 5 20 65 4 3 8 35 99
225
Minimum number of points
Multibody epipole
Recovery of epipoles
Fundamental matrices
Feature segmentation

26. Other cases: affine flows
In linear motions, geometric constraints are linear
Two-view motion constraints could be bilinear!!!
Affine motion segmentation:
constant brightness constraint
3D motion segmentation:
epipolar constraint

27. 3D motion segmentation results

28. Results

29. Results
Add details from “Experiment section” on Rene’s paper (Tech Report: A Factorization Method for 3D Multi-body Motion Estimation & Segmentation)

30. More results

31. Conclusions
There is an analytic solution to 3D motion segmentation based on
Multibody epipolar constraint: it does not depend on the segmentation of the data
Polynomial factorization: linear algebra
Solution is closed form iff n<5
A similar technique also applies to
Eigenvector segmentation: from similarity matrices
Generalized PCA: mixtures of subspaces
2-D motion segmentation: of affine motions
Future work
Reduce data complexity, sensitivity analysis, robustness

32. References
R. Vidal, Y. Ma, S. Soatto and S. Sastry Two-view multibody structure from motion, International Journal of Computer Vision, 2004
R. Vidal and S. Sastry. Optimal segmentation of dynamic scenes from two perspective views, International Conference on Computer Vision and Pattern Recognition, 2003
R. Vidal and S. Sastry. Segmentation of dynamic scenes from image intensities, IEEE Workshop on Vision and Motion Computing, 2002.

33. PRIMARY REFERENCE
A primary reference for this course is a book soon to be published by Springer Verlag, authored by me, Stefano, Jana, and Shankar Sastry. There is a bound version displayed at Springer-Verlag’s booth.