1. Bag of Visual Words for Image Representation & Visual Search Jianping Fan Dept of Computer Science UNC-Charlotte

2. 1. Interest Point Extraction & SIFT 2. Clustering for Dictionary Learning 3. Bag of Visual Words 4. Image Representation & Applications Bag-of-Visual-Words

3. Interest Point Extraction Scale-space extrema detection –Uses difference-of-Gaussian function Keypoint localization –Sub-pixel location and scale fit to a model Orientation assignment –1 or more for each keypoint Keypoint descriptor –Created from local image gradients

4. Scale space Definition: where Interest Point Extraction

5. Scale space Keypoints are detected using scale-space extrema in difference-of-Gaussian function D D definition: Efficient to compute

6. Relationship of D to Close approximation to scale-normalized Laplacian of Gaussian, Diffusion equation: Approximate ∂G/∂σ: –giving, When D has scales differing by a constant factor it already incorporates the σ 2 scale normalization required for scale-invariance

7. Interest Point Extraction

8. Difference-of-Gaussian images … first octave … … second octave … … third octave … fourth octave … …

9. Finding extrema Sample point is selected only if it is a minimum or a maximum of these points DoG scale space Extrema in this image

10. Localization 3D quadratic function is fit to the local sample points Start with Taylor expansion with sample point as the origin –where Take the derivative with respect to X, and set it to 0, giving is the location of the keypoint This is a 3x3 linear system

11. Localization Derivatives approximated by finite differences, –example: If X is > 0.5 in any dimension, process repeated

12. Filtering Contrast (use prev. equation): –If | D(X) | < 0.03, throw it out Edge-iness: –Use ratio of principal curvatures to throw out poorly defined peaks –Curvatures come from Hessian: –Ratio of Trace(H) 2 and Determinant(H) –If ratio > (r+1) 2 /(r), throw it out (SIFT uses r=10)

13. Orientation assignment Descriptor computed relative to keypoint’s orientation achieves rotation invariance Precomputed along with mag. for all levels (useful in descriptor computation) Multiple orientations assigned to keypoints from an orientation histogram –Significantly improve stability of matching

14. Keypoint images

15. Descriptor Descriptor has 3 dimensions (x,y,θ) Orientation histogram of gradient magnitudes Position and orientation of each gradient sample rotated relative to keypoint orientation

16. Descriptor Best results achieved with 4x4x8 = 128 descriptor size Normalize to unit length –Reduces effect of illumination change Cap each element to 0.2, normalize again –Reduces non-linear illumination changes –0.2 determined experimentally

17. PCA-SIFT Different descriptor (same keypoints) Apply PCA to the gradient patch Descriptor size is 20 (instead of 128) More robust, faster

18. Interest Points & SIFT Features

19. Summary Scale space Difference-of-Gaussian Localization Filtering Orientation assignment Descriptor, 128 elements

20. Dictionary Learning

27. Quantization for Identification

28. 28 Sparse Coding & Dictionary Learning Dictionary learning and sparse coding Sparse factor analysis model (Factor/feature/dish/dictionary atom) Indian Buffet process and beta process

29. Dictionary Learning

34. Image Representation via Bag-of-Visual-Words Dictionary

40. Application for Visual Search

41. How to do database indexing?

42. Application for Visual Search Visual Phrases & Contexts?

43. Multi-Resolution SIFT