A Closed Form Solution to Direct Motion Segmentation René Vidal and Dheeraj Singaraju Center for Imaging Science Johns Hopkins University

Direct 2-D motion segmentation problem Given the image intensities of a dynamic scene containing multiple motions, determine Number of motion models (translational, affine, etc.) Motion model: translational, or affine Segmentation: model to which each pixel belongs Copyright © JHU Vision Lab

Prior work on 2-D motion segmentation Probabilistic approaches (Jepson-Black’93, Ayer-Sawhney ’95, Darrel-Pentland’95, Weiss-Adelson’96, Weiss’97, Torr-Szeliski-Anandan ’99) Generative model: mixture of 2-D motion models Estimate model using Expectation Maximization E-step: Given motion models, segment image M-step: Given grouping, estimate motion models Local methods (Wang-Adelson ’93) Estimate one model per pixel using a data in a window Cluster models with K-means Aperture problem Motion across boundaries Global methods (Irani-Peleg ’92) Dominant motion: fit one motion model to all pixels Look for misaligned pixels & fit a new model to them Motion competition (Cremers’02) Piece-wise parametric model Minimize Mumford-Shah cost functional Copyright © JHU Vision Lab

Paper contributions Propose a global algebraic solution to direct 2-D motion segmentation Multibody Brightness Constancy Constraint (MBCC) 2-D translational motions and 2-D affine motions Algebraic solution to direct motion segmentation Fit MBCC linearly to image partial derivatives Compute optical flow as the derivative of the MBCC Compute motion model parameters from the cross products of the derivatives of the MBCC Main features Requires no feature tracking or correspondences Fits a mixture of motion models globally Can be used to initialize nonlinear/probabilistic methods Needs improvement Copyright © JHU Vision Lab

Brightness constancy constraint (BCC) Linear motion model: 2-D translational Bilinear motion model: 2-D affine Copyright © JHU Vision Lab

Multibody brightness constancy constraint Mixture of n 2-D motion models Multibody brightness constancy constraint Optical flow from derivative of MBCC Copyright © JHU Vision Lab

Segmentation of 2-D translational motions MBCC is homogeneous polynomial of degree n Linear on embedded data! Number of motions Veronese map Motion models Copyright © JHU Vision Lab

Segmentation of 2-D affine motions MBCC bi-homogeneous polynomials of degree n Lifting Embedding Bilinear on embedded data! Multibody motion matrix Copyright © JHU Vision Lab

Segmentation of 2-D affine motions Multibody motion can be computed linearly from Number of motion models can be computed from the rank of embedded data matrix Individual motion models can be computed from the derivatives of Copyright © JHU Vision Lab

Segmentation of 2-D affine motions If belongs to ith motion, partials of MBC at give linear combination of the rows of As 3rd row of is , 1st and 2nd rows can be obtained from cross product of partials Copyright © JHU Vision Lab

Calculation of number of motions Copyright © JHU Vision Lab

Segmentation of 2-D translational motions Copyright © JHU Vision Lab

Segmentation of 2-D translational motions Good segmentation (71% of frames) Bad segmentation (29% of frames) Copyright © JHU Vision Lab

Segmentation of 2-D affine motions Copyright © JHU Vision Lab

Segmentation of 2-D affine motions Good segmentation (78% of frames) Bad segmentation (22% of frames) Copyright © JHU Vision Lab

Optical flow for 2-D affine motions Copyright © JHU Vision Lab

Conclusions and open problems Algebraic solution to direct motion segmentation Multibody Brightness Constancy Constraint (MBCC) Optical flow & affine models: cross product of MBCC derivatives Main features Requires no feature tracking or correspondences Fits a mixture of motion models globally Can be used to initialize nonlinear/probabilistic methods Open problems How to deal with outliers? How to incorporate smoothing in space and time? How to incorporate multiple resolutions? How to segment motion models of different type? Copyright © JHU Vision Lab

Thanks

Optical flow for 2-D translational motions Copyright © JHU Vision Lab

Best segmentation results Copyright © JHU Vision Lab

Worst segmentation results Copyright © JHU Vision Lab

Best segmentation results Copyright © JHU Vision Lab

Worst segmentation results Copyright © JHU Vision Lab