1. MASKS © 2004 Invitation to 3D vision Lecture 8 Segmentation of Dynamical Scenes

2. MASKS © 2004 Invitation to 3D vision One-body two-views

3. MASKS © 2004 Invitation to 3D vision One-body multiple-views

4. MASKS © 2004 Invitation to 3D vision Motivation and problem statement A static scene: multiple 2D motion models A dynamic scene: multiple 3D motion models Given an image sequence, determine Number of motion models (affine, Euclidean, etc.) Motion model: affine (2D) or Euclidean (3D) Segmentation: model to which each pixel belongs

5. MASKS © 2004 Invitation to 3D vision Previous work on 2D motion segmentation Local methods (Wang-Adelson ’93) Estimate one model per pixel using data in a window Cluster models with K-means Iterate Aperture problem Motion across boundaries Global methods (Irani-Peleg ‘92) Dominant motion: fit one motion model to all pixels Look for misaligned pixels & fit a new model to them Iterate Normalized cuts (Shi- Malik ‘98) Similarity matrix based on motion profile Segment pixels using eigenvector

6. MASKS © 2004 Invitation to 3D vision Previous work on 3D motion segmentation Factorization techniques Orthographic/discrete: Costeira-Kanade ’98, Gear ‘98 Perspective/continuous: Vidal-Soatto-Sastry ’02 Omnidirectional/continuous: Shakernia-Vidal-Sastry ’03 Special cases: Points in a line (orth-discrete): Han and Kanade ’00 Points in a conic (perspective): Avidan-Shashua ’01 Points in a line (persp.-continuous): Levin-Shashua ’01 2-body case: Wolf-Shashua ‘01

7. MASKS © 2004 Invitation to 3D vision Previous work: probabilistic techniques Probabilistic approaches Generative model: data membership + motion model Obtain motion models using Expectation Maximization E-step: Given motion models, segment image data M-step: Given data segmentation, estimate motion models 2D Motion Segmentation Layered representation (Jepson-Black’93, Ayer-Sawhney ’95, Darrel-Pentland’95, Weiss-Adelson’96, Weiss’97, Torr-Szeliski- Anandan ’99) 3D Motion Segmentation EM+Reprojection Error: Feng-Perona’98 EM+Model Selection: Torr ’98 How to initialize iterative algorithms?

8. MASKS © 2004 Invitation to 3D vision Our approach to motion segmentation This work considers full perspective projection multiple objects general motions We show that Problem is equivalent to polynomial factorization There is a unique global closed form solution if n<5 Exact solution is obtained using linear algebra Can be used to initialize EM-based algorithms Image points Number of motions Multibody Fund. Matrix Epipolar lines Epipoles Fundamental Matrices Motion segmentation Multi epipolar lines Multi epipole

9. MASKS © 2004 Invitation to 3D vision One-dimensional segmentation Number of models?

10. MASKS © 2004 Invitation to 3D vision One-dimensional segmentation For n groups Number of groups Groups

11. MASKS © 2004 Invitation to 3D vision One-dimensional segmentation Solution is unique if Solution is closed form if and only if Solves the eigenvector segmentation problem e.g. normalized cuts

12. MASKS © 2004 Invitation to 3D vision Three-dimensional motion segmentation Generalized PCA (Vidal, Ma ’02) Solve for the roots of a polynomial of degree in one variable Solve for a linear system in variables

13. MASKS © 2004 Invitation to 3D vision The multibody epipolar constraint Rotation: Translation: Epipolar constraint Multiple motions Multibody epipolar constraint Satisfied by ALL points regardless of segmentation Segmentation is algebraically eliminated!!!

14. MASKS © 2004 Invitation to 3D vision The multibody fundamental matrix Lifting Embedding Bilinear on embedded data! Veronese map (polynomial embedding) Multibody fundamental matrix

15. MASKS © 2004 Invitation to 3D vision Estimation of the number of motions Theorem: Given image points corresponding to motions, if at least 8 points correspond to each object, then 1 243 8 3599225 Minimum number of points

16. MASKS © 2004 Invitation to 3D vision Estimation of multibody fundamental matrix 1-body motion n-body motion

17. MASKS © 2004 Invitation to 3D vision Segmentation of fundamental matrices Given rank condition for n motions linear system F Multibody epipolar transfer Multibody epipole Fundamental matrices

18. MASKS © 2004 Invitation to 3D vision Multibody epipolar transfer Multibody epipolar line Lifting Polynomial factorization

19. MASKS © 2004 Invitation to 3D vision Multibody epipole The multibody epipole is the solution of the linear system Epipoles are obtained using polynomial factorization Lifting Number of distinct epipoles

20. MASKS © 2004 Invitation to 3D vision From images to epipoles

21. MASKS © 2004 Invitation to 3D vision Fundamental matrices Columns of are epipolar lines Polynomial factorization to compute them up to scale Scales can be computed linearly

22. MASKS © 2004 Invitation to 3D vision The multibody 8-point algorithm Image point Multibody epipolar line Multibody epipolar transfer Polynomial Factorization Linear system Polynomial Factorization Epipolar lines Multibody epipole Epipoles Embedded image point Veronese map Fundamental matrix Linear system

23. MASKS © 2004 Invitation to 3D vision Remarks about the algorithm Algebraically equivalent to polynomial factorization Requires solving for roots of polynomial of degree n in one variable There is a closed form solution if n<5 The algorithm is probably polynomial time It requires O(n 4 ) image points It neglects internal structure of the multibody fundamental matrix

24. MASKS © 2004 Invitation to 3D vision Optimal 3D motion segmentation Zero-mean Gaussian noise Constrained optimization problem on Optimal function for 1 motion Optimal function for n motions Solved using Riemanian Gradien Descent

25. MASKS © 2004 Invitation to 3D vision Comparison of 1 body and n bodies

26. MASKS © 2004 Invitation to 3D vision Other cases: linearly moving objects Multibody epipole Recovery of epipoles Fundamental matrices Feature segmentation 1 2105 2 52065 1 243 8 3599225 Minimum number of points

27. MASKS © 2004 Invitation to 3D vision Other cases: affine flows Affine motion segmentation: constant brightness constraint 3D motion segmentation: epipolar constraint In linear motions, geometric constraints are linear Two-view motion constraints could be bilinear!!!

28. MASKS © 2004 Invitation to 3D vision Other cases: multiple views Compute optical flow for N pixels in F frames relative to reference frame Form optical flow matrix Similarity matrix is Apply GPCA to firstsingular vectors of to obtain the segmentation of the image using motion 0 F 2 1

29. MASKS © 2004 Invitation to 3D vision 3D motion segmentation results N = 44 + 48 + 81 = 173

30. MASKS © 2004 Invitation to 3D vision Results: affine motion segmentation

31. MASKS © 2004 Invitation to 3D vision Results

32. MASKS © 2004 Invitation to 3D vision Results

33. MASKS © 2004 Invitation to 3D vision More results

34. MASKS © 2004 Invitation to 3D vision Conclusions There is an analytic solution to 3D motion segmentation based on Multibody epipolar constraint: it does not depend on the segmentation of the data Polynomial factorization: linear algebra Solution is closed form iff n<5 A similar technique also applies to Eigenvector segmentation: from similarity matrices Generalized PCA: mixtures of subspaces 2-D motion segmentation: of affine motions Future work Reduce data complexity, sensitivity analysis, robustness

35. MASKS © 2004 Invitation to 3D vision References R. Vidal, Y. Ma, S. Soatto and S. Sastry. Two-view multibody structure from motion, International Journal of Computer Vision, 2004 R. Vidal and S. Sastry. Optimal segmentation of dynamic scenes from two perspective views, International Conference on Computer Vision and Pattern Recognition, 2003 R. Vidal and S. Sastry. Segmentation of dynamic scenes from image intensities, IEEE Workshop on Vision and Motion Computing, 2002.