Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto

Hierarchical Models Manually built hierarchical model proposed by Marr And Nishihara (“Representation and recognition of the spatial organization of three dimensional shapes”, Proc. of Royal Soc. of London, 1978)

Our goal Automatically construct a generic hierarchical shape model from exemplars Challenges:  Cannot assume similar appearance among different exemplars  Generic features are highly ambiguous  Generic features may not be in one-to-one correspondence

Automatically constructed Hierarchical Models Input: Question: What is it? Output:

Stages of the system Exemplar images Extract Blob Graphs Match Blob Graphs (many-to-many) Extract Parts Extract Decomposition Relations Extract Attachment Relations Assemble Final Model Blob graphs Many-to-many correspondences Model parts Model decomposition relations Model attachment relations

Blob Graph Construction Exemplar images Extract Blob Graphs Match Blob Graphs (many-to-many) Extract Parts Extract Decomposition Relations Extract Attachment Relations Assemble Final Model Blob graphs Many-to-many correspondences Model parts Model decomposition relations Model attachment relations

Blob Graph Construction  Edges are invariant to articulation  Choose the largest connected component. On the Representation and Matching of Qualitative Shape at Multiple Scales A. Shokoufandeh, S. Dickinson, C. Jonsson, L. Bretzner, and T. Lindeberg, ECCV 2002

Feature matching Exemplar images Extract Blob Graphs Match Blob Graphs (many-to-many) Extract Parts Extract Decomposition Relations Extract Attachment Relations Assemble Final Model Blob graphs Many-to-many correspondences Model parts Model decomposition relations Model attachment relations

Feature matching One-to-one matching. Rely on shape and context, not appearance! ? Many-to-many matching

Feature embedding and EMD Spectral embedding

Returning to our set of inputs  Many-to-many matching of every pair of exemplars.

Part Extraction Exemplar images Extract Blob Graphs Match Blob Graphs (many-to-many) Extract Parts Extract Decomposition Relations Extract Attachment Relations Assemble Final Model Blob graphs Many-to-many correspondences Model parts Model decomposition relations Model attachment relations

Results of the part extraction stage

What is next?

Extracting attachment relations Exemplar images Extract Blob Graphs Match Blob Graphs (many-to-many) Extract Parts Extract Decomposition Relations Extract Attachment Relations Assemble Final Model Blob graphs Many-to-many correspondences Model parts Model decomposition relations Model attachment relations

Extracting attachment relations Right arm is typically connected to torso in exemplar images !

Extracting decomposition relations Exemplar images Extract Blob Graphs Match Blob Graphs (many-to-many) Extract Parts Extract Decomposition Relations Extract Attachment Relations Assemble Final Model Blob graphs Many-to-many correspondences Model parts Model decomposition relations Model attachment relations

Extracting decomposition relations

Model construction stage summary Model Construction:  Clustering blobs into parts based on one-to-one matching results.  Recovering relations between parts based on individual matching and attachment results.

Assemble Final Model Exemplar images Extract Blob Graphs Match Blob Graphs (many-to-many) Extract Parts Extract Decomposition Relations Extract Attachment Relations Assemble Final Model Blob graphs Many-to-many correspondences Model parts Model decomposition relations Model attachment relations

Conclusions General framework for constructing a generic decompositional model from different exemplars with dissimilar appearance. General framework for constructing a generic decompositional model from different exemplars with dissimilar appearance. Recovering decompositional relations requires solving the difficult many-to-many graph matching problem. Recovering decompositional relations requires solving the difficult many-to-many graph matching problem. Preliminary results indicate good model recovery from noisy features. Preliminary results indicate good model recovery from noisy features.

Future work Construct models for objects other than humans. Construct models for objects other than humans. Provide scale invariance during matching. Provide scale invariance during matching. Automatically learn perceptual grouping relations from labeled examples. Automatically learn perceptual grouping relations from labeled examples. Develop indexing and matching framework for decompositional models. Develop indexing and matching framework for decompositional models.