1. A feature-based kernel for object classification P. Moreels - J-Y Bouguet Intel

2. One scenario for image query : Relevance Feedback Relevance Feedback ImageDatabase Query + “Baby with house” SimilarityMetricCandidateResults Final Result SVM classifier ???

3. Outline Motivation Features and distances Starting point: the pyramid match kernel Extension : the max-kernel. Experiments Conclusions

4. Similarity Metric between pairs of images Need to derive a measure of similarity between images How similar are those two images?

5. Similarity Metric between pairs of images Need to derive a measure of similarity between images How similar are those two images? It is desired that:

6. Image descriptors: features Smaller volume of information Robustness to deformations (lighting, rotation, affine transformations) Detector: difference-of-gaussians Descriptor: SIFT = 128D collection of local gradients

7. Match-based Similarity IDEA: First establish correspondence between the two sets of points and then compute a “distance” metric based on the matching result STEP 1: Establish Correspondence STEP 2: Compute similarity PROBLEM: Until now, no matching-based image similarity metric has been shown to satisfy the MERCER conditions

8. Our main contribution Derive a new image similarity metric that is based on point correspondence and satisfies the Mercer condition Methodoly: generalize another metric developed by Grauman at MIT (pyramid kernel) while preserving its Mercer quality

9. Pyramid match – K.Grauman (ICCV05) Only appearance is considered Image represented in terms of multi-scale histograms Appearance space

10. Matching process Soft matches by histogram intersection Fine resolution to coarse resolution More weight at fine resolution: 2 -level =1/size(bin) Level 0 Level 1 Level 2

11. Final score (Kernel) count intersection at current level discards matches already counted at previous levels - Fine resolution first - Coarse resolution last This kernel verifies Mercer condition ! (Odone et all, TIP, 2005) more weight given to best matches

12. Issues should be matched at this level level 0 level 1 level 2 level 3 counted only here Boundary problems x 2

13. Issues level n 2 level =size(bin) approximates poorly the distance between 2 points Weight function f(d)= 1/d over- emphasizes small distances w = weight = 1/size(bin) c = correct weight = 1/d (w-c)/c

14. From discrete to continous 2 k is a poor approximation  increase the number of resolution steps Boundary problems  use translation of bins level 0 level 1 level 2 level 3 Verifies Mercer condition  (1+  ) 11 00 22 33

15. STEP 1: Establish Correspondence Our kernel This kernel is easy to compute Uses exact distances No over-emphasis of low distances still verifies Mercer condition best matches first where

16. Experiments – distance accuracy Random sets of 2D points Compares distances based on: our kernel, pyramid match, Earth Mover’s Distance (EMD = optimal solution to the matching problem, based on simplex) Distances measured between simulated images for pyramid match, max-kernel and EMD Corresponding probability density function

17. Caltech database – 101 categories

18. Some data ~50 to ~300 images per class Performance of the competitors: –Chance : 1% –Fei-Fei & Perona : 16% –Berg & Malik : 48% –Holub & Perona : 40% –Grauman & Darrell : 43% Classification using a SVM

19. Classification results category vs. bg:  performance = 89% 10 random categories  performance: 61%

20. Classification results 7 good, 7 bad categories Performance: 45%

21. Computation time 100-300 features per image 0.1s to compute kernel value b/w 2 images Training: each classifier (100 classifiers) requires 50-100 x 50-100 kernel values SVM optimization is not an issue Testing: most training samples are SVMs  of the order of 30 x 80 kernel values per class Parallelization: everything is computed independently –in the training phase –in the test phase

22. Conclusions The MaxKernel is more accurate than pyramid match, more practical than EMD Good approximation of the optimal distance Verifies Mercer  SVM classification OK Initial classification performance in same ballpark as the competition TODO: add some geometry – e.g. Hough transform to filter out wrong correspondences