Coherent Laplacian 3D protrusion segmentation Oxford Brookes Vision Group Queen Mary, University of London, 11/12/2009 Fabio Cuzzolin

The problem to recognize actions we need to extract features segmenting moving articulated 3D bodies into parts along sequences, in a consistent way in an unsupervised fashion robustly, with respect to changes of the topology of the moving body as a building block of a wider motion analysis and capture framework ICCV-HM'07, CVPR'08, to submit to IJCV

Coherent 3D Laplacian protrusion segmentation Laplacian methods Locally linear embedding An unsupervised algorithm Results on real sequences Results on synthetic sequences Topology changes and missing data Comparisons Influence of parameters Applications

Spectral methods given a dataset of points {X i, i=1,..,N} compute an affinity matrix A(i,j) = d(X i,X j ) apply SVD to this affinity matrix this yields a list of eigenvalues and associated eigenvectors a number of eigenvectors are selected, and used to build an embedded cloud of points {Y i, i=1,..,N}

Laplacian methods the affinity matrix is the Laplacian operator, or some function of it Graph Laplacian: operator on functions f defined on sets X of points of the form L[f] i = j N(i) w ij (f i f j ) maps each such function f to another function L[f] N(i) is the set of neighbors of X i f i is the value of the function f on X i

Laplacian eigenfunctions Laplacian eigenfunctions/values have nice topological properties eigenvalues are invariant for volume-preserving transformations eigenfunctions for a base for all functions on X their zero-level sets are related to protrusions and symmetries of the underlying cloud

Coherent 3D Laplacian protrusion segmentation Laplacian methods Locally linear embedding An unsupervised algorithm Results on real sequences Results on synthetic sequences Topology changes and missing data Comparisons Influence of parameters Applications

Locally Linear Embedding for each data point we compute the weights W ij that best reconstruct X i from its neighbors: argmin W i |X i j W ij X j | 2 Low-dim embeddings Y_i are obtained by argmin Y i |Y i j W ij Y j | 2 i.e., local neighbors are the same, subject to affinity matrix M = (I W) T (I W) optimal embedding bottom d+1 eigevectors (but last one)

LLE algorithm

Protrusion preservation protrusions are high-curvature surface regions by definition LLE leaves unchanged the weights of each neighborhood (the affine coordinates of X_i in the base of its neighbors) weights depend on pairwise distances between points preserving weights means preserving distances up to a scale this happens in surface neighborhoods too if they are all roughly the same size curvature distribution is preserved: protrusions!

Lower dimensionality protrusion preservation is an effect of local isometry (the first constraint of LLE) the covariance constraint has the effect of producing a lower-dimensional embedded cloud LLE is a constrained minimization problem in physics constraints G(X)=0 are associated with a force orthogonal to the constraint surface the covariance constraint is associated with a force that pulls the cloud of points outward, reducing the chain of neighborhoods to a string

Clustering in the embedding space Locally Linear Embedding: preserves the local structure of the dataset generates a lower-dim embedded cloud preserves protrusions less sensitive to topology changes than other methods

Pose invariance with LLE generates a lower-dim, widely separated embedded cloud less sensitive to topology changes than other methods less expensive then ISOMAP (refs. Jenkins, Chellappa) local neighbourhoods stable under articulated motion

Coherent 3D Laplacian protrusion segmentation Laplacian methods Locally linear embedding An unsupervised algorithm Results on real sequences Results on synthetic sequences Topology changes and missing data Comparisons Influence of parameters Applications

Algorithm due to their low dimensionality, protrusions are detected in the embedding space they can be clustered as sets of collinear points using k- wise clustering segmentation is brought back to 3D

K-wise clustering LLE maps the 3D shape to a lower-dimensional shape Idea: clustering collinear points together K-wise clustering: K-wise clustering: a hypergraph H is built by measuring the affinity of all triads a weighted graph G which approximates H is constructed by constrained linear least square optimization the approximating graph is partitioned by spectral clustering (n-cut)

Protrusion detection protrusions can be easily detected after embedding due to low dimensionality an embedded point is a termination if its projection on the line interpolating its neighborhood is an extremum

Seed propagation along time To ensure time consistency clusters seeds have to be propagated along time Old positions of clusters in 3D are added to new cloud and embedded Result: new seeds

Merging/splitting clusters At each t all branch terminations of Y(t) are detected; if t=0 they are used as seeds for k-wise clustering; otherwise (t>0) standard k-means is performed on Y(t) using branch terminations as seeds, yielding a rough partition of the embedded cloud into distinct branches; propagated seeds in the same partition are merged; for each partition of Y(t) not containing any old seed a new seed is defined as the related branch termination.

Coherent 3D Laplacian protrusion segmentation Laplacian methods Locally linear embedding An unsupervised algorithm Results on real sequences Results on synthetic sequences Topology changes and missing data Comparisons Influence of parameters Applications

Results on real sequences 1 for real sequences ground truth is difficult to gather we can still visually appreciate the quality and consistency of the resulting segmentation

Results on real sequences 2 for real sequences ground truth is difficult to gather we can still visually appreciate the quality and consistency of the resulting segmentation

Coherent 3D Laplacian protrusion segmentation Laplacian methods Locally linear embedding An unsupervised algorithm Results on real sequences Results on synthetic sequences Topology changes and missing data Comparisons Influence of parameters Applications

Ground truth for synthetic data For synthetic sequences of sequences for which pose has been estimated, ground truth can be gathered performance indicators: compare the obtained segmentation with the three natural ones on the right

Scores for synthetic sequences

Coherent 3D Laplacian protrusion segmentation Laplacian methods Locally linear embedding An unsupervised algorithm Results on real sequences Results on synthetic sequences Topology changes and missing data Comparisons Influence of parameters Applications

Handling topology changes when topology changes occur cluters merge and/or split to accommodate them

Handling missing data

Coherent 3D Laplacian protrusion segmentation Laplacian methods Locally linear embedding An unsupervised algorithm Results on real sequences Results on synthetic sequences Topology changes and missing data Comparisons Influence of parameters Applications

Vs EM clustering EM clustering fits a multi-Gaussian distribution to the data through the EM algorithm number of cluster is automatically estimated Left: our algo; right: EM clustering

Vs ISOMAP the same propagation scheme can be applied in the ISOMAP space extremely sensitive to topology changes ISOMAP computes an embedding by applying MDS to the affinity matrix of all pairwise geodesic distances

Performance comparison segmentation scores for two other real sequences solid: our method; dashed: EM; dashdot: ISOMAP

Coherent 3D Laplacian protrusion segmentation Laplacian methods Locally linear embedding An unsupervised algorithm Results on real sequences Results on synthetic sequences Topology changes and missing data Comparisons Influence of parameters Applications

Estimating neighborhood size the number of neighbors can be estimated from the data sequence admissible k: yields neighborhoods which do not span different bodyparts

Eigenvector selection according to the eigenvectors we select after decomposition, we obtain different unsupervised segmentations

Stability with respect to parameter values consistency, segmentation and average scores for 2 sequences, as a function of parameter values k and d

Coherent 3D Laplacian protrusion segmentation Laplacian methods Locally linear embedding An unsupervised algorithm Results on real sequences Results on synthetic sequences Topology changes and missing data Comparisons Influence of parameters Applications

Model recovery example a sequence representing a counting hand is portrait along time, the algorithm learns the object is formed by more and more rigid segments

Laplacian matching of dense meshes or voxelsets as embeddings are pose-invariant (for articulated bodies) they can then be used to match dense shapes by simply aligning their images after embedding ICCV '07 – NTRL, ICCV '07 – 3dRR, CVPR '08, to submit to PAMI

Eigenfunction Histogram assignment Algorithm: compute Laplacian embedding of the two shapes find assignment between eigenfunctions of the two shapes this selects a section of the embedding space embeddings are orthogonally aligned there by EM

Results Appls: graph matching, protein analysis, motion capture To propagate bodypart segmentation in time Motion field estimation, action segmentation

Conclusions Unsupervised bodypart segmentation algorithm which ensure consistency along time Spectral method: clustering is performed in the embedding space (in particular after LLE) as shape becomes lower-dim and different bodyparts are widely separated Seeds are propagated along time and merged/splitted according to topology variations Compares favorably with other techniques First block of motion analysis framework (matching, action recognition, etc.)