1. 1 Part 1: Classical Image Classification Methods Kai Yu Dept. of Media Analytics NEC Laboratories America Andrew Ng Computer Science Dept. Stanford University

2. Outline of Part 2 5/15/2015 2 Local Features, Sampling, Visual Words Discriminative Methods - Bag-of-Words (BoW) representation - Spatial pyramid matching (SPM) Generative Methods - Part-based methods - Topic models Local Features, Sampling, Visual Words Discriminative Methods - Bag-of-Words (BoW) representation - Spatial pyramid matching (SPM) Generative Methods - Part-based methods - Topic models

3. Outline of Part 2 5/15/2015 3 Local Features, Sampling, Visual Words Discriminative Methods - Bag-of-Words (BoW) representation - Spatial pyramid matching (SPM) Generative Methods - Part-based methods - Topic models Local Features, Sampling, Visual Words Discriminative Methods - Bag-of-Words (BoW) representation - Spatial pyramid matching (SPM) Generative Methods - Part-based methods - Topic models

4. Local features 5/15/2015 4 Distinctive descriptors of local image patches Invariant to local translation, scale, … and sometimes rotation or general affine transformations The most famous choice is the SIFT feature Distinctive descriptors of local image patches Invariant to local translation, scale, … and sometimes rotation or general affine transformations The most famous choice is the SIFT feature

5. Sampling local features from images 5/15/2015 5 A set of points Image credits: F-F. Li, E. Nowak, J. Sivic

6. Visual words 5/15/2015 6 Similar points are grouped into one visual word Algorithms: k-means, agglomerative clustering, … Points from different images are then more easily compared. Similar points are grouped into one visual word Algorithms: k-means, agglomerative clustering, … Points from different images are then more easily compared. Slide credit: Kristen Grauman

7. Outline of Part 2 5/15/2015 7 Local Features, Sampling, Visual Words, … Discriminative Methods - Bag-of-Words (BoW) representation - Spatial pyramid matching (SPM) Generative Methods - Part-based methods - Topic models Local Features, Sampling, Visual Words, … Discriminative Methods - Bag-of-Words (BoW) representation - Spatial pyramid matching (SPM) Generative Methods - Part-based methods - Topic models

8. Bag-of-words (BoW) representation 5/15/2015 8 Analogy to documents Adapted from tutorial slides by Fei-Fei et al.

9. BoW for object categorization 5/15/2015 9 Works pretty well for whole-image classification Slide credit: Svetlana Lazebnik Csurka et al. (2004), Willamowski et al. (2005), Grauman & Darrell (2005), Sivic et al. (2003, 2005)

10. Unsupervised Dictionary Learning 5/15/2015 10 image database Sample local features from images Run k-mean or other clustering algorithm to get dictionary Dictionary is also called “codebook” Sample local features from images Run k-mean or other clustering algorithm to get dictionary Dictionary is also called “codebook” SIFT space R1 R2 R3

11. Compute BoW histogram for each image 5/15/2015 11 R1 R2 R3 Assign sift features into clusters Compute the frequency of each cluster within an image R1 R2 R3 BoW histogram representations

12. Indication of BoW histogram 5/15/2015 12 Summarize entire image based on its distribution of visual word occurrences Turn bags of different sizes into a fixed length vector Analogous to bag of words representation commonly used for text categorization. Summarize entire image based on its distribution of visual word occurrences Turn bags of different sizes into a fixed length vector Analogous to bag of words representation commonly used for text categorization.

13. Image classification based on BoW histogram 5/15/2015 13 dog bird Decision boundary BoW histogram vector space Learn a classification model to determine the decision boundary Nonlinear SVMs are commonly applied. Learn a classification model to determine the decision boundary Nonlinear SVMs are commonly applied.

14. Issues 5/15/2015 14 Sampling strategy Learning codebook: size? supervised?, … Classification: which method? scalability? Scalability: how to handle millions of data? How to use spatial information? Sampling strategy Learning codebook: size? supervised?, … Classification: which method? scalability? Scalability: how to handle millions of data? How to use spatial information?

15. Spatial information 5/15/2015 15 The BoW removes spatial layout. This increases the invariance to scale, translation, and deformation, But sacrifices discriminative power, especially when the spatial layout is important. The BoW removes spatial layout. This increases the invariance to scale, translation, and deformation, But sacrifices discriminative power, especially when the spatial layout is important. Slide adapted from Bill Freeman

16. Spatial pyramid matching 5/15/2015 16 Compute BoW for image regions at different locations in various scales Figure credit: Svetlana Lazebnik

17. A common pipeline for discriminative image classification using BoW 5/15/2015 17 K-means Dense/Sparse SIFT dictionary Dictionary Learning VQ Coding Dense/Sparse SIFT Spatial Pyramid Pooling Nonlinear SVM Image Classification

18. Combining multiple descriptors 5/15/2015 18 Multiple Feature Detectors Multiple Descriptors: SIFT, shape, color, … VQ Coding and Spatial Pooling Nonlinear SVM Diagram from SurreyUVA_SRKDA, winner team in PASCAL VOC 2008

19. Outline of Part 2 5/15/2015 19 Local Features, Sampling, Visual Words, … Discriminative Methods - Bag-of-Words (BoW) representation - Spatial pyramid matching (SPM) Generative Methods - Part-based methods - Topic models Local Features, Sampling, Visual Words, … Discriminative Methods - Bag-of-Words (BoW) representation - Spatial pyramid matching (SPM) Generative Methods - Part-based methods - Topic models

20. 5/15/2015 20 Topic models for images w N c z D  Latent Dirichlet Allocation (LDA) Fei-Fei et al. ICCV 2005 “beach” Slide credit Fei-Fei Li

21. Part-based Model 5/15/2015 21 Fischler & Elschlager 1973 Rob Fergus ICCV09 Tutorial

22. For a comprehensive coverage of object categorization models, please visit 5/15/2015 22 Recognizing and Learning Object Categories Li Fei-Fei (Stanford), Rob Fergus (NYU), Antonio Torralba (MIT) http://people.csail.mit.edu/torralba/shortCourseRLOC/