Robust spectral 3D-bodypart segmentation along time Fabio Cuzzolin, Diana Mateus, Edmond Boyer, Radu Horaud Perception project meeting 24/4/2007 Submitted to ICCV07

Robust spectral segmentation Consistent bodypart segmentation in sequences Why clustering in the embedding space K-wise clustering Branch detection Seed propagation Merging-splitting clusters Algorithm Results Influence of d and K Topology changes Comparison with EM clustering Comparison with ISOMAP clustering

Problem Segmenting bodyparts of moving articulated bodies along sequences, in a consistent way in an unsupervised fashion robustly, with respect to changes of the topology of the moving body as a bulding block of a wider motion analysis and capture framework

Clustering in the embedding space Locally Linear Embedding: preserves the local structure of the dataset Locally Linear Embedding: preserves the local structure of the dataset generates a lower-dim embedded cloud less sensitive to topology changes than other methods shape of the embedded cloud fairly stable under AND less computationally expensive then ISOMAP

Pose invariance with LLE To ensure consistent segmentation the stability of the embedded cloud is necessary To ensure consistent segmentation the stability of the embedded cloud is necessary LLE works with local neighborhoods -> stable under articulated motion LLE works with local neighborhoods -> stable under articulated motion

Algorithm Pictorial illustration of the overall algorithm

K-wise clustering LLE maps the 3D shape to a lower-dimensional shape LLE maps the 3D shape to a lower-dimensional shape Idea: clustering collinear points together Idea: clustering collinear points together K-wise clustering: K-wise clustering: a hypergraph H is built by measuring the affinity of all triads a weighted graph G which approximates H is constructed by constrained linear least square optimization the approximating graph is partitioned by spectral clustering (n-cut)

Branch detection and number of clusters Branches can be detected easily Branches can be detected easily An embedded point is a termination if its projection on the line interpolating its neighborhood is an extremum An embedded point is a termination if its projection on the line interpolating its neighborhood is an extremum

Seed propagation along time To ensure time consistency clusters seeds have to be propagated along time To ensure time consistency clusters seeds have to be propagated along time Old positions of clusters in 3D are added to new cloud and embedded Old positions of clusters in 3D are added to new cloud and embedded Result: new seeds Result: new seeds

Merging/splitting clusters 1. At each t all branch terminations of Y(t) are detected; 2. if t=0 they are used as seeds for k-wise clustering; 3. otherwise (t>0) standard k-means is performed on Y(t) using branch terminations as seeds, yielding a rough partition of the embedded cloud into distinct branches; 4. propagated seeds in the same partition are merged; 5. for each partition of Y(t) not containing any old seed a new seed is defined as the related branch termination.

Results - 1

Results - 2

Estimating k The number of neighbors can be estimated from the data sequence The number of neighbors can be estimated from the data sequence Admissible k: yields neighborhoods which do not span different bodyparts

Influence of dimension Choosing a larger dimension for the embedding space improves the resolution of the segmentation Choosing a larger dimension for the embedding space improves the resolution of the segmentation

Change of topology When topology changes, clusters merge or split to adapt When topology changes, clusters merge or split to adapt

Performance of EM clustering EM clustering fits a multi-Gaussian distribution to the data through the EM algorithm EM clustering fits a multi-Gaussian distribution to the data through the EM algorithm

Performance of ISOMAP The same propagation scheme can be applied in the ISOMAP space The same propagation scheme can be applied in the ISOMAP space extremely sensitive to topology changes extremely sensitive to topology changes

Conclusions Unsupervised bodypart segmentation algorithm which ensure consistency along time Unsupervised bodypart segmentation algorithm which ensure consistency along time Spectral method: clustering is performed in the embedding space (in particular after LLE) as shape becomes lower-dim and different bodyparts are widely separated Seeds are propagated along time and merged/splitted according to topology variations Compares favorably with other techniques First step of motion analysis (matching, action recognition, etc.)