Motion Segmentation at Any Speed Shrinivas J. Pundlik Department of Electrical and Computer Engineering, Clemson University, Clemson, SC

Gestalt Theory and Visual Perception The human visual system: Focus on well organized patterns rather than disparate parts “Grouping” - the idea behind visual perception Factors affecting the grouping process: proximity, similarity, closure, smoothness, symmetry, common fate and so on.

Gestalt Laws of Grouping Basis of many image and video segmentation algorithms Work well in combinations. For example, proximity and similarity Motion segmentation – grouping or aggregating entities with common fate ProximityContinuityCommon Fate

Applications of Motion Segmentation Object detection and tracking Surveillance Robot Motion Image and Video Compression Video Editing/Motion Magnification Shape Recovery

Existing Approaches Extraction of Motion Layers Wang and Adelson 1994, Weiss 1996 Ayer and Sawhney 1995,Xiao and Shah 2005 Ke and Kanade 2002 Detecting Motion Discontinuities Black and Fleet 1999, Birchfield 1998 Normalized Cuts Shi and Malik 1998 Feature Point Grouping Beymer 1997, Fua 2003 Kanhere et. al. 2005

Preview

Incremental Motion Segmentation Existing approaches Consider 2 frames or a spatio-temporal volume Threshold on velocities τ > (Δx/ Δt) Proposed approach An incremental approach Threshold on position Waits till enough evidence accumulates before segmenting

Different Approaches To Segmentation Segmentation/Grouping AgglomerativeDivisive Start with single point and grow the group. Region growing or Region merging Start with entire data and split into clusters. Clustering or partitioning seed 1 seed 2 first step last step first step last step

Representation of Motion Why use feature points instead of optic flow? Reduced time and complexity of computation Reliable and repeatable Well suited for tracking over long sequences

Feature Tracking Idea behind feature tracking: minimize the dissimilarity between two feature windows in the successive frames Good Features: Small image regions having high intensity variation in more than one direction Affine Consistency Check:

Feature Clustering Clustering Data: Feature displacement over multiple frames K-means clustering by fitting lines Works better than clustering points

Results of Feature Clustering by K-means Limitation: Clustering not accurate for more challenging sequences

Affine Partitioning Requires prior initialization and number of groups to be found Processing only on feature motion between two frames

Normalized Cuts Graph: G(V,E) Partitions: A,B Weight of an edge: w Affinity Matrix: Feature motion between two frames

Region Growing Process over two frames Select seed point Fit affine model to neighbors Repeat until the group does not change: Discard all features except the one near the centroid Grow group by including neighboring features with similar motion till it grows no further Update the affine model

Finding Neighbors Traditional way: Spatial window Makes the algorithm sensitive to the feature locations Alternative: Delaunay Traingulation Simple and efficient technique

Finding Consistent groups Parameters affecting region growing: grouping threshold, choice of frames and seed point Different choice of seed points produce different grouping results Features grouped together irrespective of the choice of seed points are consistent feature groups

Consistent Groups

Maintaining Groups Over Time Finding new feature groups Segmenting new objects entering the scene Splitting existing feature groups Split when configuration of a group changes over time Adding new features to existing feature groups Include new scene information over time

Results Segmentation results for the statue sequence

Results Frames 3, 4, 5, 6 of the statue sequence with threshold = 0.7 Frames 8, 64, 188, 395,6 of the fast statue sequence (generated by dropping every alternate frame)

Results Segmentation results for different sequences

Conclusions Segmentation based on the availability of evidence An incremental approach – able to handle long sequences