A feature-based kernel for object classification P. Moreels - J-Y Bouguet Intel

One scenario for image query : Relevance Feedback Relevance Feedback ImageDatabase Query + “Baby with house” SimilarityMetricCandidateResults Final Result SVM classifier ???

Outline Motivation Features and distances Starting point: the pyramid match kernel Extension : the max-kernel. Experiments Conclusions

Similarity Metric between pairs of images Need to derive a measure of similarity between images How similar are those two images?

Similarity Metric between pairs of images Need to derive a measure of similarity between images How similar are those two images? It is desired that:

Image descriptors: features Smaller volume of information Robustness to deformations (lighting, rotation, affine transformations) Detector: difference-of-gaussians Descriptor: SIFT = 128D collection of local gradients

Match-based Similarity IDEA: First establish correspondence between the two sets of points and then compute a “distance” metric based on the matching result STEP 1: Establish Correspondence STEP 2: Compute similarity PROBLEM: Until now, no matching-based image similarity metric has been shown to satisfy the MERCER conditions

Our main contribution Derive a new image similarity metric that is based on point correspondence and satisfies the Mercer condition Methodoly: generalize another metric developed by Grauman at MIT (pyramid kernel) while preserving its Mercer quality

Pyramid match – K.Grauman (ICCV05) Only appearance is considered Image represented in terms of multi-scale histograms Appearance space

Matching process Soft matches by histogram intersection Fine resolution to coarse resolution More weight at fine resolution: 2 -level =1/size(bin) Level 0 Level 1 Level 2

Final score (Kernel) count intersection at current level discards matches already counted at previous levels - Fine resolution first - Coarse resolution last This kernel verifies Mercer condition ! (Odone et all, TIP, 2005) more weight given to best matches

Issues should be matched at this level level 0 level 1 level 2 level 3 counted only here Boundary problems x 2

Issues level n 2 level =size(bin) approximates poorly the distance between 2 points Weight function f(d)= 1/d over- emphasizes small distances w = weight = 1/size(bin) c = correct weight = 1/d (w-c)/c

From discrete to continous 2 k is a poor approximation  increase the number of resolution steps Boundary problems  use translation of bins level 0 level 1 level 2 level 3 Verifies Mercer condition  (1+  ) 11 00 22 33

STEP 1: Establish Correspondence Our kernel This kernel is easy to compute Uses exact distances No over-emphasis of low distances still verifies Mercer condition best matches first where

Experiments – distance accuracy Random sets of 2D points Compares distances based on: our kernel, pyramid match, Earth Mover’s Distance (EMD = optimal solution to the matching problem, based on simplex) Distances measured between simulated images for pyramid match, max-kernel and EMD Corresponding probability density function

Caltech database – 101 categories

Some data ~50 to ~300 images per class Performance of the competitors: –Chance : 1% –Fei-Fei & Perona : 16% –Berg & Malik : 48% –Holub & Perona : 40% –Grauman & Darrell : 43% Classification using a SVM

Classification results category vs. bg:  performance = 89% 10 random categories  performance: 61%

Classification results 7 good, 7 bad categories Performance: 45%

Computation time features per image 0.1s to compute kernel value b/w 2 images Training: each classifier (100 classifiers) requires x kernel values SVM optimization is not an issue Testing: most training samples are SVMs  of the order of 30 x 80 kernel values per class Parallelization: everything is computed independently –in the training phase –in the test phase

Conclusions The MaxKernel is more accurate than pyramid match, more practical than EMD Good approximation of the optimal distance Verifies Mercer  SVM classification OK Initial classification performance in same ballpark as the competition TODO: add some geometry – e.g. Hough transform to filter out wrong correspondences