Segmentation of Dynamic Scenes René Vidal Dept. of EECS, UC Berkeley Yi Ma (UIUC) Stefano Soatto (UCLA) Shankar Sastry (UCB)

Outline Data segmentation: linear case One dimension: eigenvector segmentation K-dimensions: Generalized PCA Motion segmentation: bilinear case 3D Motion Segmentation Affine Motion Segmentation Examples and Control Applications

Motivation Given a set of image points obtain: Number of independently moving objects Segmentation: object to which each point belongs Motion: rotation and translation of each object Structure: depth of each point

Previous Work 2-D motion segmentation Look for flow discontinuities Apply normalized cuts to similarity matrix Model selection and EM 3-D geometric motion segmentation Orthographic factorization: Costeira & Kanade ’98 Points in a conic: Avidan-Shashua ’01 Han and Kanade ’00 Points in a line: Levin-Shashua ‘01 2-body motions: Wolf-Shashua ‘01

Our Approach to Segmentation We propose an algebraic geometric approach to data segmentation Number of groups = degree of a polynomial Groups ≈ roots of a polynomial = polynomial factorization We show that There exist segmentation independent constraints There exists a unique closed form solution if n<5 The exact solution can be computed using linear algebra

One-dimensional Segmentation

One-dimensional Segmentation For n groups Number of groups Groups

One-dimensional Segmentation Solution is unique if Solution is closed form if Solves the eigenvector segmentation problem e.g. normalized cuts

K-dimensional Segmentation: Generalized PCA via Polynomial Factorization Data lies on K-1 dimensional subspaces Generalized PCA (Vidal-Ma-Sastry ‘02) Solve for the roots of a polynomial of degree n in one variable Solve for a linear system in n variables

Motion Segmentation: 2 views Two-view motion constraints are bilinear Affine motion segmentation: constant brightness constraint 3D motion segmentation: epipolar constraint

Motion Segmentation: 2 views Multibody affine and epipolar constraints

Multibody Affine and Fundamental Matrix Lifting Embedding Multibody epipolar constraint

Multibody Affine and Fundamental Matrix 1-body motion n-body motion

Number of Independent Motions Theorem: Given image points corresponding to motions, if at least 8 (6) points correspond to each object, then Minimum number of points Affine motion 1 2 4 3 6 24 64 160 8 35 99 225 3D motion

Multibody epipolar transfer 3D Motion Segmentation Given rank condition n linear system F Multibody epipolar transfer Multibody epipole Fundamental matrices

3D Motions: Multibody epipolar transfer Lifting Multibody epipolar line Polynomial factorization

3D Motions: Multibody epipole Lifting The multibody epipole is the solution of the linear system Number of distinct epipoles Epipoles are obtained using polynomial factorization

3D Motions: Fundamental matrices Columns of are epipolar lines Polynomial factorization to compute them up to scale Scales can be computed linearly

Multibody SFM Algorithm Image point Veronese map Embedded image point Multibody epipolar transfer Multibody epipolar line Polynomial Factorization Epipolar lines Linear system Multibody epipole Polynomial Factorization Epipoles Linear system Fundamental matrix

Affine Motion Segmentation Given rank condition n linear system A Affine Matrices 2 Rows and 1 Column Polynomial Factorization

Affine Motion Segmentation Affine motion constraint Nonlinear algebraic error

Example 1: 3D Motion Segmentation

Example 2: Affine Motion Segmentation

Example 3: Affine Motion Segmentation

Example 4: Multiple View Segmentation

Experimental Results Add details from “Experiment section” on Rene’s paper (Tech Report: A Factorization Method for 3D Multi-body Motion Estimation & Segmentation)

Experimental Results

Example 5: Omnidirectional Segmentation Shakernia-Vidal-Sastry (Workshop on Motion, 2002)

Applications Vision-based landing (Shakernia-Vidal-Sharp-Ma-Sastry, ICRA 2002)

Applications Pursuit-evasion games (Vidal-Shakernia-Kim- Shim-Sastry, Trans. on Robotics & Automation 2002)

Applications Formation control Vidal-Shakernia-Sastry, 2002

Conclusions There is an algebraic/geometric solution to Data segmentation: linear constraints Motion segmentation: bilinear constraints Solution based on Polynomial factorization: linear algebra Solution is closed form if n<5 Showed applications in Pursuit-evasion games & Formation control

Linearly moving objects 1 2 10 5 20 65 4 3 8 35 99 225 Minimum number of points Multibody epipole Recovery of epipoles Fundamental matrices Feature segmentation