1. Beyond bags of features: Part-based models Many slides adapted from Fei-Fei Li, Rob Fergus, and Antonio Torralba

2. Implicit shape models Visual codebook is used to index votes for object position B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004Combined Object Categorization and Segmentation with an Implicit Shape Model training image annotated with object localization info visual codeword with displacement vectors

3. Implicit shape models Visual codebook is used to index votes for object position B. Leibe, A. Leonardis, and B. Schiele, Combined Object Categorization and Segmentation with an Implicit Shape Model, ECCV Workshop on Statistical Learning in Computer Vision 2004Combined Object Categorization and Segmentation with an Implicit Shape Model test image

4. Implicit shape models: Training 1.Build codebook of patches around extracted interest points using clustering

5. Implicit shape models: Training 1.Build codebook of patches around extracted interest points using clustering 2.Map the patch around each interest point to closest codebook entry

6. Implicit shape models: Training 1.Build codebook of patches around extracted interest points using clustering 2.Map the patch around each interest point to closest codebook entry 3.For each codebook entry, store all positions it was found, relative to object center

7. Implicit shape models: Testing 1.Given test image, extract patches, match to codebook entry 2.Cast votes for possible positions of object center 3.Search for maxima in voting space 4.Extract weighted segmentation mask based on stored masks for the codebook occurrences

8. Source: B. Leibe Original image Example: Results on Cows

9. Interest points Example: Results on Cows Source: B. Leibe

10. Example: Results on Cows Matched patches Source: B. Leibe

11. Example: Results on Cows Probabilistic votes Source: B. Leibe

12. Example: Results on Cows Hypothesis 1 Source: B. Leibe

13. Example: Results on Cows Hypothesis 2 Source: B. Leibe

14. Example: Results on Cows Hypothesis 3 Source: B. Leibe

15. Additional examples B. Leibe, A. Leonardis, and B. Schiele, Robust Object Detection with Interleaved Categorization and Segmentation, IJCV 77 (1-3), pp. 259-289, 2008.Robust Object Detection with Interleaved Categorization and Segmentation

16. Generative part-based models R. Fergus, P. Perona and A. Zisserman, Object Class Recognition by Unsupervised Scale-Invariant Learning, CVPR 2003Object Class Recognition by Unsupervised Scale-Invariant Learning

17. Model the probability of an image given a class What is a generative model? vs.

18. Maximum a posteriori (MAP) decision: assign the image to the class that gets the highest posterior probability P(class | image) Making a decision

19. Maximum a posteriori (MAP) decision: assign the image to the class that gets the highest posterior probability P(class | image) Bayes rule: Making a decision

20. Generative vs. discriminative models Discriminative methods: model posterior Generative methods: model likelihood and prior posterior likelihood prior

21. Generative vs. discriminative models GenerativeDiscriminative Class densities Posterior probabilities

22. The Naïve Bayes model Generative model for a bag of features Assume that each feature f i is conditionally independent given the class c:

23. Generative part-based model Candidate parts Part descriptors Part locations

24. Generative part-based model h: assignment of features to parts Part 2 Part 3 Part 1

25. Generative part-based model h: assignment of features to parts Part 2 Part 3 Part 1

26. Generative part-based model High-dimensional appearance space Distribution over patch descriptors

27. Generative part-based model 2D image space Distribution over joint part positions

28. Results: Faces Face shape model Patch appearance model Recognition results

29. Results: Motorbikes and airplanes

30. Pictorial structures P. Felzenszwalb and D. Huttenlocher, Pictorial Structures for Object Recognition, IJCV 61(1), 2005Pictorial Structures for Object Recognition Set of parts (oriented rectangles) connected by edges Recognition problem: find the most probable part layout l 1, …, l n in the image

31. Pictorial structures MAP formulation: maximize posterior Energy-based formulation: minimize minus the log of probability: AppearanceGeometry Matching cost Deformation cost

32. Pictorial structures: Complexity Suppose there are n parts, and each has h possible positions in the image What is the complexity of finding the optimal layout? Brute force search: O(h n ) Tree-structured model: O(h 2 n) Deformation cost is quadratic: O(hn)