Segmentation of Dynamic Scenes from Image Intensities René Vidal Shankar Sastry Department of EECS, UC Berkeley

Motivation and problem statement A static scene: multiple 2D motion models A dynamic scene: multiple 3D motion models Given an image sequence, determine Number of motion models (affine, Euclidean, etc.) Motion model: affine (2D) or Euclidean (3D) Segmentation: model to which each pixel belongs 3-D Motion Segmentation (Vidal-Soatto-Ma-Sastry, ECCV’02) Generalization of the 8-point algorithm Multibody fundamental matrix This Talk: 2-D Motion Segmentation

Previous work Probabilistic approaches (Jepson-Black’93, Ayer-Sawhney ’95, Darrel-Pentland’95, Weiss-Adelson’96, Weiss’97, Torr-Szeliski-Anandan ’99) Generative model: data membership + motion model Obtain motion models using Expectation Maximization E-step: Given motion models, segment image data M-step: Given data segmentation, estimate motion models How to initialize iterative algorithms? Spectral clustering: normalized cuts (Shi-Malik ‘98) Similarity matrix based on motion profile Local methods (Wang-Adelson ’94) Estimate one model per pixel using a data in a window Global methods (Irani-Peleg ‘92) Dominant motion: fit one motion model to all pixels Look for misaligned pixels & fit a new model to them

Our Approach to Motion Segmentation Can we estimate ALL motion models simultaneously using ALL the image measurements? When can we do so analytically? In closed form? Is there a formula for the number of models? We propose an algebraic geometric approach to affine motion segmentation Number of models = degree of a polynomial Groups ≈ roots of a polynomial = polynomial factorization In the absence of noise Derive a constraint that is independent on the segmentation There exists a unique solution which is closed form iff n<5 The exact solution can be computed using linear algebra In the presence of noise Derive a maximum likelihood algorithm for zero-mean Gaussian noise in which the E-step is algebraically eliminated

One-dimensional Segmentation with a. ames Number of models?

One-dimensional Segmentation with a. ames For n groups Number of groups Groups

Motion segmentation: the affine model Constant brightness constraint Affine motion model for the optical flow Bilinear affine constraint Mixture of n affine motion models

The multibody affine constraint 1-dimensional case Multibody affine constraint Veronese map Affine segmentation case

The multibody affine matrix Multibody affine constraint Multibody affine matrix Embedding Lifting Embedding

Affine motion segmentation algorithm 1-dimensional case Affine segmentation case Estimate all models: coefficients of a polynomial Estimate number models: rank of a matrix Chance a by c Change \ell by \b and transpose Np ng Estimate individual models: roots/factors of the polynomial

Estimation of individual affine models Can be reduced to scalar case!! Factorization of affine motion models Factorization of bilinear forms can be reduced to factorization of linear forms Factorization of linear forms corresponds to segmentation of mixtures of subspaces Generalized PCA: mixture of subspaces Find roots of polynomial of degree n in one variable Solve one linear systems in n variables

Optimal affine motion segmentation Zero-mean Gaussian noise Minimize distance error in image intensities subject to multibody affine constraints Using Langrange optimization After some algebra

Experimental results: flower sequence

Experimental results Two motions Camera panning to the right Car translating to the right http://www.cs.otago.ac.nz/research/vision/Research/

Conclusions There is an analytic solution to affine motion segmentation based on Multibody affine constraint: segmentation independent Polynomial factorization: linear algebra Solution is closed form iff n<5 A similar technique also applies to Eigenvector segmentation: from similarity matrices Generalized PCA: mixtures of subspaces 3-D motion segmentation: of fundamental matrices Future work Reduce data complexity, sensitivity analysis, robustness