The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features Kristen Grauman Trevor Darrell MIT

Sets of features

examples under varying conditions local shape features invariant region descriptors

Problem How to build a discriminative classifier using the set representation? Kernel-based methods (e.g. SVM) are appealing for efficiency and generalization power… But what is an appropriate kernel? Each instance is unordered set of vectors Varying number of vectors per instance

Existing set kernels Fit (parametric) model to each set, compare with distance over models Kondor & Jebara, Moreno et al., Lafferty & Lebanon, Cuturi & Vert, Wolf & Shashua Restrictive assumptions Ignoring set statistics Compute pair-wise similarity between all vectors in each set Wallraven et al., Lyu, Boughhorbel et al. General family of algebraic functions combining local (vector) kernels Shashua & Hazan High complexity

Partial matching for sets of features Compare sets by computing a partial matching between their features. Robust to clutter, segmentation errors, occlusion…

Pyramid match optimal partial matching

Pyramid match overview Place multi-dimensional, multi-resolution grid over point sets Consider points matched at finest resolution where they fall into same grid cell Approximate similarity between matched points with worst case similarity at given level No explicit search for matches! Pyramid match kernel measures similarity of a partial matching between two sets:

Pyramid match kernel Number of newly matched pairs at level i Measure of difficulty of a match at level i Approximate partial match similarity

Feature extraction, Histogram pyramid: level i has bins of size 2 i

Counting matches Histogram intersection

Counting new matches Difference in histogram intersections across levels counts number of new pairs matched matches at this levelmatches at previous level Histogram intersection

Pyramid match kernel Weights inversely proportional to bin size Normalize kernel values to avoid favoring large sets measure of difficulty of a match at level i histogram pyramids number of newly matched pairs at level i

Efficiency For sets with m features of dimension d, and pyramids with L levels, computational complexity of Pyramid match kernel: Existing set kernel approaches: or

Example pyramid match Level 0

Example pyramid match Level 1

Example pyramid match Level 2

Example pyramid match pyramid match optimal match

100 sets with 2D points, cardinalities vary between 5 and 100 Trial number (sorted by optimal distance) [Indyk & Thaper] Matching output Approximation of the optimal partial matching

Building a classifier Train SVM by computing kernel values between all labeled training examples Classify novel examples by computing kernel values against support vectors One-versus-all for multi-class classification Convergence is guaranteed since pyramid match kernel is positive-definite.

Object recognition results ETH-80 database 8 object classes Features: –Harris detector –PCA-SIFT descriptor, d=10 KernelComplexityRecognition rate Match [Wallraven et al.] 84% Bhattacharyya affinity [Kondor & Jebara] 85% Pyramid match 84% Eichhorn and Chapelle 2004

Object recognition results Caltech objects database 101 object classes Features: –SIFT detector –PCA-SIFT descriptor, d=10 30 training images / class 43% recognition rate (1% chance performance) seconds per match

Localization Inspect intersections to obtain correspondences between features Higher confidence correspondences at finer resolution levels observationtarget

Pyramid match regression Pose estimation from contour features Train SVR with CG data Features: shape context histograms

Summary: Pyramid match kernel optimal partial matching between sets of features number of new matches at level idifficulty of a match at level i

Summary: Pyramid match kernel A new similarity measure based on implicit correspondences that approximates the optimal partial matching linear time complexity no independence assumption model-free insensitive to clutter positive-definite function fast, effective object recognition

Future work Geometric constraints Fast search of large databases with the pyramid match for image retrieval Use as a filter for a slower, explicit correspondence method Alternative feature types and classification domains