1. Computer Vision Group University of California Berkeley Shape Matching and Object Recognition using Shape Contexts Jitendra Malik U.C. Berkeley (joint work with S. Belongie, J. Puzicha, G. Mori)

2. Computer Vision Group University of California Berkeley Outline Shape matching and isolated object recognition Scaling up to general object recognition

3. Computer Vision Group University of California Berkeley Biological Shape D’Arcy Thompson: On Growth and Form, 1917 –studied transformations between shapes of organisms

4. Computer Vision Group University of California Berkeley Deformable Templates: Related Work Fischler & Elschlager (1973) Grenander et al. (1991) Yuille (1991) von der Malsburg (1993)

5. Computer Vision Group University of California Berkeley Matching Framework Find correspondences between points on shape Estimate transformation Measure similarity modeltarget...

6. Computer Vision Group University of California Berkeley Comparing Pointsets

7. Computer Vision Group University of California Berkeley Shape Context Count the number of points inside each bin, e.g.: Count = 4 Count = 10... FCompact representation of distribution of points relative to each point

8. Computer Vision Group University of California Berkeley Shape Context

9. Computer Vision Group University of California Berkeley Shape Contexts Invariant under translation and scale Can be made invariant to rotation by using local tangent orientation frame Tolerant to small affine distortion –Log-polar bins make spatial blur proportional to r Cf. Spin Images (Johnson & Hebert) - range image registration

10. Computer Vision Group University of California Berkeley Comparing Shape Contexts Compute matching costs using Chi Squared distance: Recover correspondences by solving linear assignment problem with costs C ij [Jonker & Volgenant 1987]

11. Computer Vision Group University of California Berkeley Matching Framework Find correspondences between points on shape Estimate transformation Measure similarity modeltarget...

12. Computer Vision Group University of California Berkeley 2D counterpart to cubic spline: Minimizes bending energy: Solve by inverting linear system Can be regularized when data is inexact Thin Plate Spline Model Duchon (1977), Meinguet (1979), Wahba (1991)

13. Computer Vision Group University of California Berkeley Matching Example modeltarget

14. Computer Vision Group University of California Berkeley Outlier Test Example

15. Computer Vision Group University of California Berkeley Synthetic Test Results Fish - deformation + noiseFish - deformation + outliers ICPShape ContextChui & Rangarajan

16. Computer Vision Group University of California Berkeley Matching Framework Find correspondences between points on shape Estimate transformation Measure similarity modeltarget...

17. Computer Vision Group University of California Berkeley Terms in Similarity Score Shape Context difference Local Image appearance difference –orientation –gray-level correlation in Gaussian window –… (many more possible) Bending energy

18. Computer Vision Group University of California Berkeley Object Recognition Experiments Kimia silhouette dataset Handwritten digits COIL 3D objects (Nayar-Murase) Human body configurations Trademarks

19. Computer Vision Group University of California Berkeley Shape Similarity: Kimia dataset

20. Computer Vision Group University of California Berkeley Quantitative Comparison rank Number correct

21. Computer Vision Group University of California Berkeley Handwritten Digit Recognition MNIST 60 000: –linear: 12.0% –40 PCA+ quad: 3.3% –1000 RBF +linear: 3.6% –K-NN: 5% –K-NN (deskewed) : 2.4% –K-NN (tangent dist.) : 1.1% –SVM: 1.1% –LeNet 5: 0.95% MNIST 600 000 (distortions): –LeNet 5: 0.8% –SVM: 0.8% –Boosted LeNet 4: 0.7% MNIST 20 000: –K-NN, Shape Context matching: 0.63%

22. Computer Vision Group University of California Berkeley

23. Computer Vision Group University of California Berkeley Results: Digit Recognition 1-NN classifier using: Shape context + 0.3 * bending + 1.6 * image appearance

24. Computer Vision Group University of California Berkeley COIL Object Database

25. Computer Vision Group University of California Berkeley Error vs. Number of Views

26. Computer Vision Group University of California Berkeley Prototypes Selected for 2 Categories Details in Belongie, Malik & Puzicha (NIPS2000)

27. Computer Vision Group University of California Berkeley Editing: K-medoids Input: similarity matrix Select: K prototypes Minimize: mean distance to nearest prototype Algorithm: –iterative –split cluster with most errors Result: Adaptive distribution of resources (cfr. aspect graphs)

28. Computer Vision Group University of California Berkeley Error vs. Number of Views

29. Computer Vision Group University of California Berkeley Human body configurations

30. Computer Vision Group University of California Berkeley Automatically Locating Keypoints User marks keypoints on exemplars Find correspondence with test shape Transfer keypoint position from exemplar to the test shape.

31. Computer Vision Group University of California Berkeley Results

32. Computer Vision Group University of California Berkeley Trademark Similarity

33. Computer Vision Group University of California Berkeley Outline Shape matching and isolated object recognition Scaling up to general object recognition –Many objects (Mori, Belongie & Malik, CVPR 01) –Gray scale matching (Berg & Malik, CVPR 01) –Objects in scenes (scanning or segmentation)

34. Computer Vision Group University of California Berkeley Mori, Belongie, Malik (CVPR 01) Fast Pruning –Given a query shape, quickly return a shortlist of candidate matches –Database of known objects will be large: ~30000 Detailed Matching –Perform computationally expensive comparisons on only the few shapes in the shortlist

35. Computer Vision Group University of California Berkeley Representative Shape Contexts Match using only a few shape contexts –Don’t need to compare every one

36. Computer Vision Group University of California Berkeley Snodgrass Results

37. Computer Vision Group University of California Berkeley Results

38. Computer Vision Group University of California Berkeley Conclusion Introduced new matching algorithm matching based on shape contexts and TPS Robust to outliers & noise Forms basis of object recognition technique that performs well in a variety of domains using exactly the same algorithm