Shape Matching and Object Recognition using Low Distortion Correspondence Alexander C. Berg, Tamara L. Berg, Jitendra Malik U.C. Berkeley

Object Category Recognition

Deformable Template Matching with Exemplars for Recognition Use exemplars as deformable templates Find a correspondence between the query image and each template Query Image Database of Templates

Deformable Template Matching with Exemplars for Recognition Use exemplars as deformable templates Find a correspondence between the query image and each template Query Image Database of Templates Best matching template is a helicopter

Correspondence for Deformable Template Matching Evaluate correspondence based on: – Similarity of appearance near feature points – Similarity in configuration of the feature points Query Template

Correspondence for Deformable Template Matching Evaluate correspondence based on: – Similarity of appearance near feature points – Similarity in configuration of the feature points Query Template

Correspondence for Deformable Template Matching Evaluate correspondence based on: – Similarity of appearance near feature points – Similarity in configuration of the feature points Query Template

Correspondence for Deformable Template Matching Evaluate correspondence based on: – Similarity of appearance near feature points – Similarity in configuration of the feature points Query Template

Correspondence Result Query Template

Interpolated Correspondence Using Thin Plate Splines Query Template

Correspondence for Deformable Template Matching Query Template r ij r i'j'

Geometric Blur (Local Appearance Descriptor) Geometric Blur Descriptor ~ Compute sparse channels from image Extract a patch in each channel Apply spatially varying blur and sub-sample (Idealized signal) Descriptor is robust to small affine distortions Berg & Malik '01

Geometric Blur (Local Appearance Similarity) Geometric Blur Descriptor Geometric Blur Descriptor ~

Are Features Enough? Not Quite... Color indicates similarity using Geometric Blur Descriptor

Measuring Distortion (Similarity in Configuration) Query Template R ij S i'j' Measure distortion in vectors between pairs of feature points - R and S same length for rotations - R and S same direction for scalings

Cost Function as IQP Appearance cost if i -> j Distortion cost if i -> j and k -> l Integer Quadratic Programming Problem... iff template point i maps to query point j, If binary vector x represents a correspondence cf. Maciel & Costeira '03

Optimization Integer Quadratic Programming is NP hard The instances we generate seem easy Using a linear bound to initialize gradient descent provides good results – (In fact better than the guarantee of Goemans & Williamson's randomized algorithm) Varying the linear constraints on x allows – one-one, one-many, or fixed number of outliers, etc.

Correspondence Result

Interpolated Correspondence Using Thin Plate Splines

Quadratic Assignment (Using IQP)

Linear Assignment (e.g. Hungarian)

Correspondence Examples (Shape Matching)

Application to Recognition Caltech 101 – 101 classes of objects + background Large Scale Roughly aligned Large intra-class variation Fei-Fei, Fergus, Perona '04

Caltech 101 Recognition Results Chance~1% N.N. whole image16% Discriminative version of Constellation Model27% N.N. Geometric Blur Descriptors38% Low Distortion Correspondence (GB+IQP)45% 102 way Alternative Forced Choice test (15 training examples per class) 102 way confusion matrix 100% 0%

Model Building for Segmentation Rough correspondence to each example image Average quality of alignment Threshhold

Automatic vs Hand Segmentation

Application to Recognition Faces Face dataset from Berg et al '03 – Medium to large scale faces – AP News photos 20 face exemplars Same methodology as Caltech 101, but multiple objects / image – After one face is identified its features are removed and the search continues Compared to a detector from Mikolajczyk based on Schneiderman & Kanade, that is quite successful on this dataset

Application to Recognition Faces

Conclusion Use rich descriptors that are insensitive to typical transformations – Geometric Blur Enforce relationship constraints among corresponding features – Integer Quadratic Programming Estimate smooth transform – Thin Plate Splines

Thank You Acknowledgements Charless Fowlkes Xiaofeng Ren David Forsyth