1. Image Classification using Sparse Coding: Advanced Topics
Part 3:
Image Classification using Sparse Coding: Advanced Topics
Kai Yu
Dept. of Media Analytics
NEC Laboratories America
Andrew Ng
Computer Science Dept.
Stanford University

2. Outline of Part 3 Why can sparse coding learn good features?
Intuition, topic model view, and geometric view
A theoretical framework: local coordinate coding
Two practical coding methods
Recent advances in sparse coding for image classification
4/19/2017

3. Outline of Part 3 Why can sparse coding learn good features?
Intuition, topic model view, and geometric view
A theoretical framework: local coordinate coding
Two practical coding methods
Recent advances in sparse coding for image classification
4/19/2017

4. Intuition: why sparse coding helps classification?
Figure from
The coding is a nonlinear feature mapping
Represent data in a higher dimensional space
Sparsity makes prominent patterns more distinctive
4/19/2017

5. A “topic model” view to sparse coding
Basis 1
Basis 2
Both figures adapted from CVPR10 tutorial by F. Bach, J. Mairal, J. Ponce and G. Sapiro
Each basis is a “direction” or a “topic”.
Sparsity: each datum is a linear combination of only a few bases.
Applicable to image denoising, inpainting, and super-resolution.
4/19/2017

6. A geometric view to sparse coding
Data manifold
Data
Basis
Each basis is somewhat like a pseudo data point – “anchor point”
Sparsity: each datum is a sparse combination of neighbor anchors.
The coding scheme explores the manifold structure of data.
4/19/2017

7. MNIST Experiment: Classification using SC
Try different values
60K training, 10K for test
Let k=512
Linear SVM on sparse codes
4/19/2017

8. MNIST Experiment: Lambda = 0.0005
Each basis is like a part or direction.
4/19/2017

9. MNIST Experiment: Lambda = 0.005
Again, each basis is like a part or direction.
4/19/2017

10. MNIST Experiment: Lambda = 0.05
Now, each basis is more like a digit !
4/19/2017

11. MNIST Experiment: Lambda = 0.5
Like clustering now!
4/19/2017

12. Geometric view of sparse coding
Error: 4.54%
Error: 3.75%
Error: 2.64%
When SC achieves the best classification accuracy, the learned bases are like digits – each basis has a clear local class association.
Implication: exploring data geometry may be useful for classification.
4/19/2017

13. Distribution of coefficients (MNIST)
Neighbor bases tend to get nonzero coefficients
Let’s further check what’s happening when best classification performance is achieved.
4/19/2017

14. Distribution of coefficient (SIFT, Caltech101)
Similar observation here!
4/19/2017

15. Recap: two different views to sparse coding
Discover “topic” components
Each basis is a “direction”
Sparsity: each datum is a linear combination of several bases.
Related to topic model
View 2
Geometric structure of data manifold
Each basis is an “anchor point”
Sparsity: each datum is a linear combination of neighbor anchors.
Somewhat like a soft VQ (link to BoW)
Either can be valid for sparse coding under certain circumstances.
View 2 seems to be helpful to sensory data classification.
4/19/2017

16. Outline of Part 3 Why can sparse coding learn good features?
Intuition, topic model view, and geometric view
A theoretical framework: local coordinate coding
Two practical coding methods
Recent advances in sparse coding for image classification
4/19/2017

17. Key theoretical question
Feature learning
Classification
Why unsupervised feature learning via sparse coding can help classification?
4/19/2017

18. The image classification setting for analysis
Sparse Coding
Dense local feature
Linear Pooling
Linear SVM
Implication:
Learning an image classifier is a matter of learning nonlinear functions on patches.
Function on patches
Function on images

19. Illustration: nonlinear learning via local coding
locally linear
data points
bases
4/19/2017

20. How to learn a nonlinear function?
Step 1: Learning the dictionary from unlabeled data
4/19/2017

21. How to learn a nonlinear function?
Step 2: Use the dictionary to encode data
4/19/2017

22. How to learn a nonlinear function?
Step 3: Estimate parameters
Global linear weights to be learned
Sparse codes of data
Nonlinear local learning via learning a global linear function.
4/19/2017

23. A good coding scheme should 1. have a small coding error,
Local Coordinate Coding (LCC): connect coding to nonlinear function learning
Yu et al NIPS-09
If f(x) is (alpha, beta)-Lipschitz smooth
The key message:
A good coding scheme should
1. have a small coding error,
2. and also be sufficiently local
Function approximation error
Coding error
Locality term
4/19/2017

24. Outline of Part 3 Why can sparse coding learn good features?
Intuition, topic model view, and geometric view
A theoretical framework: local coordinate coding
Two practical coding methods
Recent advances in sparse coding for image classification
4/19/2017

25. Application of LCC theory
Fast Implementation with a large dictionary
A simple geometric way to improve BoW
Wang et al, CVPR 10
Zhou et al, ECCV 10
4/19/2017

26. Application of LCC theory
Fast Implementation with a large dictionary
A simple geometric way to improve BoW
4/19/2017

27. The larger dictionary, the higher accuracy, but also the higher computation cost
Yu et al NIPS-09
Yang et al CVPR 09
The same observation for Caltech-256, PASCAL, ImageNet, …
4/19/2017

28. Locality-constrained linear coding a fast implementation of LCC
Wang et al, CVPR 10
Dictionary Learning: k-means (or hierarchical k-means)
Coding for X,
Step 1 – ensure locality: find the K nearest bases
Step 2 – ensure low coding error:
4/19/2017

29. Competitive in accuracy, cheap in computation
Comparable with sparse coding
This is one of the two major algorithms applied by NEC-UIUC team to achieve the No.1 position in ImageNet challenge 2010!
Sparse coding
Significantly better than sparse coding
Wang et al CVPR 10
4/19/2017

30. Application of the LCC theory
Fast Implementation with a large dictionary
A simple geometric way to improve BoW
4/19/2017

31. Interpret “BoW + linear classifier”
Piece-wise local constant (zero-order)
data points
cluster centers

32. Super-vector coding: a simple geometric way to improve BoW (VQ)
Zhou et al, ECCV 10
Piecewise local linear (first-order)
Local tangent
data points
cluster centers

33. Super-vector coding: a simple geometric way to improve BoW (VQ)
If f(x) is beta-Lipschitz smooth, and
Local tangent
Function approximation error
Quantization error
4/19/2017

34. Super-vector coding: learning nonlinear function via a global linear model
Let be the VQ coding of
Super-vector codes of data
This is one of the two major algorithms applied by NEC-UIUC team to achieve the No.1 position in PASCAL VOC 2009!
Global linear weights to be learned
4/19/2017

35. Summary of Geometric Coding Methods
Vector Quantization
(BoW)
(Fast) Local Coordinate Coding
Super-vector Coding
All lead to higher-dimensional, sparse, and localized coding
All explore geometric structure of data
New coding methods are suitable for linear classifiers.
Their implementations are quite straightforward.

36. Things not covered here
Improved LCC using Local Tangent, Yu & Zhang, ICML10
Mixture of Sparse Coding, Yang et al ECCV 10
Deep Coding Network, Lin et al NIPS 10
Pooling methods
Max-pooling works well in practice, but appears to be ad-hoc.
An interesting analysis on max-pooling, Boureau et al. ICML 2010
We are working on a linear pooling method, which has a similar effect as max-pooling. Some preliminary results already in the super-vector coding paper, Zhou et al, ECCV2010.
4/19/2017

37. Outline of Part 3 Why can sparse coding learn good features?
Intuition, topic model view, and geometric view
A theoretical framework: local coordinate coding
Two practical coding methods
Recent advances in sparse coding for image classification
4/19/2017

38. Fast approximation of sparse coding via neural networks
Gregor & LeCun, ICML-10
The method aims at improving sparse coding speed in coding time, not training speed, potentially make sparse coding practical for video.
Idea: Given a trained sparse coding model, use its input outputs as training data to train a feed-forward model
They showed a speedup of X20 faster. But not evaluated on real video data.
4/19/2017

39. Group sparse coding
Bengio et al, NIPS 09
Sparse coding is on patches, the image representation is unlikely sparse.
Idea: enforce joint sparsity via L1/L2 norm on sparse codes of a group of patches.
The resultant image representation becomes sparse, which can save the memory cost, but the classification accuracy decreases.
4/19/2017

40. Learning hierarchical dictionary
Jenatton, Mairal, Obozinski, and Bach, 2010
A node can be active only if its ancestors are active.
4/19/2017

41. Reference
Image Classification using Super-Vector Coding of Local Image Descriptors, Xi Zhou, Kai Yu, Tong Zhang, and Thomas Huang. In ECCV 2010.
Efficient Highly Over-Complete Sparse Coding using a Mixture Model, Jianchao Yang, Kai Yu, and Thomas Huang. In ECCV 2010.
Learning Fast Approximations of Sparse Coding, Karol Gregor and Yann LeCun. In ICML 2010.
Improved Local Coordinate Coding using Local Tangents, Kai Yu and Tong Zhang. In ICML 2010.
Sparse Coding and Dictionary Learning for Image Analysis, Francis Bach,  Julien Mairal, Jean Ponce, and Guillermo Sapiro. CVPR 2010 Tutorial
Supervised translation-invariant sparse coding, Jianchao Yang, Kai Yu, and Thomas Huang, In CVPR 2010.
Learning locality-constrained linear coding for image classification, Jingjun Wang, Jianchao Yang, Kai Yu, Fengjun Lv, Thomas Huang, and Yihong Gong. In CVPR 2010.
Group Sparse Coding, Samy Bengio, Fernando Pereira, Yoram Singer, and Dennis  Strelow, In NIPS*2009.
Nonlinear learning using local coordinate coding, Kai Yu, Tong Zhang, and Yihong Gong. In NIPS*2009.
Linear spatial pyramid matching using sparse coding for image classification, Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. In CVPR 2009.
Efficient sparse coding algorithms. Honglak Lee, Alexis Battle, Raina Rajat and Andrew Y.Ng. In NIPS*2007.
4/19/2017