1. Sparse Coding and Its Extensions for Visual Recognition
Kai Yu
Media Analytics Department
NEC Labs America, Cupertino, CA

2. Visual Recognition is HOT in Computer Vision
80 Million Tiny Images
Caltech 101
ImageNet
PASCAL VOC
3/27/2017

3. The pipeline of machine visual perception
Most Efforts in Machine Learning
Low-level sensing
Pre-processing
Feature extract.
Feature selection
Inference: prediction, recognition
Most critical for accuracy
Account for most of the computation
Most time-consuming in development cycle
Often hand-craft in practice
3/27/2017

4. Computer vision features
SIFT
Spin image
HoG
RIFT
GLOH
Slide Credit: Andrew Ng

5. Learning everything from data
Machine Learning
Low-level sensing
Pre-processing
Feature extract.
Feature selection
Inference: prediction, recognition
Machine Learning
3/27/2017

6. BoW + SPM Kernel
Bag-of-visual-words representation (BoW) based on vector quantization (VQ)
Spatial pyramid matching (SPM) kernel
Combining multiple features, this method had been the state-of-the-art on Caltech-101, PASCAL, 15 Scene Categories, …
3/27/2017
Figure credit: Fei-Fei Li, Svetlana Lazebnik

7. Winning Method in PASCAL VOC before 2009
VQ Coding, Histogram, SPM
Multiple Feature Sampling Methods
Multiple Visual Descriptors
Nonlinear SVM
3/27/2017

8. Convolution Neural Networks
Conv. Filtering Pooling Conv. Filtering Pooling
The architectures of some successful methods are not so much different from CNNs

9. BoW+SPM: the same architecture
e.g, SIFT, HOG
VQ Coding
Average Pooling
(obtain histogram)
Nonlinear SVM
Local Gradients
Pooling
Observations:
Nonlinear SVM is not scalable
VQ coding may be too coarse
Average pooling is not optimal
Why not learn the whole thing?

10. Develop better methods
Better Coding
Better Pooling
Scalable Linear Classifier

11. Sparse Coding
Sparse coding (Olshausen & Field,1996). Originally developed to explain early visual processing in the brain (edge detection).
Training: given a set of random patches x, learning a dictionary of bases [Φ1, Φ2, …]
Coding: for data vector x, solve LASSO to find the sparse coefficient vector a
3/27/2017

12. Sparse Coding Example x » 0.8 * f36 + 0.3 * f42 + 0.5 * f63
Natural Images
Learned bases (f1 , …, f64): “Edges”
Test example
» 0.8 * * *
x » 0.8 * f * f * f63
[a1, …, a64] = [0, 0, …, 0, 0.8, 0, …, 0, 0.3, 0, …, 0, 0.5, 0]
(feature representation)
Compact & easily interpretable
Slide credit: Andrew Ng

13. … Self-taught Learning Motorcycles Not motorcycles Unlabeled images
[Raina, Lee, Battle, Packer & Ng, ICML 07]
Motorcycles
Not motorcycles
Unlabeled images …
Testing:
What is this?
Testing:
What is this?
Slide credit: Andrew Ng

14. Classification Result on Caltech 101
9K images, 101 classes
64%
SIFT VQ + Nonlinear SVM
50%
Pixel Sparse Coding + Linear SVM
3/27/2017

15. Scalable Linear Classifier
Sparse Coding on SIFT
[Yang, Yu, Gong & Huang, CVPR09]
Sparse Coding
Max Pooling
Scalable Linear Classifier
Local Gradients
Pooling
e.g, SIFT, HOG

16. 64% 73% Sparse Coding on SIFT Caltech-101 SIFT VQ + Nonlinear SVM
[Yang, Yu, Gong & Huang, CVPR09]
Caltech-101
64%
SIFT VQ + Nonlinear SVM
73%
SIFT Sparse Coding + Linear SVM
3/27/2017

17. Scalable Linear Classifier
What we have learned?
Sparse Coding
Max Pooling
Scalable Linear Classifier
Local Gradients
Pooling
e.g, SIFT, HOG
Sparse coding is a useful stuff (why?)
Hierarchical architecture is needed

18. MNIST Experiments
Error: 4.54%
Error: 3.75%
Error: 2.64%
When SC achieves the best classification accuracy, the learned bases are like digits – each basis has a clear local class association.
3/27/2017

19. Distribution of coefficient (SIFT, Caltech101)
Neighbor bases tend to get nonzero coefficients
3/27/2017

20. Geometry of data manifold
Interpretation 1
Discover subspaces
Each basis is a “direction”
Sparsity: each datum is a linear combination of only several bases.
Related to topic model
Interpretation 2
Geometry of data manifold
Each basis an “anchor point”
Sparsity is induced by locality: each datum is a linear combination of neighbor anchors.
3/27/2017

21. A Function Approximation View to Coding
Setting: f(x) is a nonlinear feature extraction function on image patches x
Coding: nonlinear mapping
x  a
typically, a is high-dim & sparse
Nonlinear Learning:
f(x) = <w, a>
A coding scheme is good if it helps learning f(x)
3/27/2017

22. A Function Approximation View to Coding – The General Formulation
Function Approx. Error
An unsupervised learning objective ≤
3/27/2017

23. Local Coordinate Coding (LCC)
Yu, Zhang & Gong, NIPS 09
Wang, Yang, Yu, Lv, Huang CVPR 10
Dictionary Learning: k-means (or hierarchical k-means)
Coding for x, to obtain its sparse representation a
Step 1 – ensure locality: find the K nearest bases
Step 2 – ensure low coding error:
3/27/2017

24. Super-Vector Coding (SVC)
Zhou, Yu, Zhang, and Huang, ECCV 10
Dictionary Learning: k-means (or hierarchical k-means)
Coding for x, to obtain its sparse representation a
Step 1 – find the nearest basis of x, obtain its VQ coding
e.g. [0, 0, 1, 0, …]
Step 2 – form super vector coding:
e.g. [0, 0, 1, 0, …, 0, 0, (x-m3），0，…]
Zero-order
Local tangent
3/27/2017

25. Function Approximation based on LCC
Yu, Zhang, Gong, NIPS 10
locally linear
data points
bases
3/27/2017

26. Function Approximation based on SVC
Zhou, Yu, Zhang, and Huang, ECCV 10
Piecewise local linear (first-order)
Local tangent
data points
cluster centers

27. PASCAL VOC Challenge 2009 No.1 for 18 of 20 categories
Ours
Best of
Other Teams
Difference
Classes
No.1 for 18 of 20 categories
We used only HOG feature on gray images
3/27/2017

28. 1.4 million images, 1000 classes,
ImageNet Challenge 2010
1.4 million images, 1000 classes,
top5 hit rate
~40%
VQ + Intersection Kernel
64%~73%
Various Coding Methods + Linear SVM
50%
Classification accuracy
3/27/2017

29. Hierarchical sparse coding
Yu, Lin, & Lafferty, CVPR 11
Learning from unlabeled data
Conv. Filtering Pooling Conv. Filtering Pooling

30. A two-layer sparse coding formulation
3/27/2017

31. MNIST Results -- classification
 HSC vs. CNN: HSC provide even better performance than CNN
 more amazingly, HSC learns features in unsupervised manner!

32. MNIST results -- effect of hierarchical learning
Comparing the Fisher score of HSC and SC
 Discriminative power: is significantly improved by HSC although HSC is unsupervised coding

33. MNIST results -- learned codebook
One dimension in the second layer: invariance to translation, rotation, and deformation

34. Caltech101 results -- classification
 Learned descriptor: performs slightly better than SIFT + SC

35. Caltech101 results -- learned codebook
 First layer bases: very much like edge detectors.

36. Conclusion and Future Work
“function approximation” view to derive novel sparse coding methods.
Locality – one way to achieve sparsity and it’s really useful. But we need deeper understanding of the feature learning methods
Interesting directions
Hierarchical coding – Deep Learning (many papers now!)
Faster methods for sparse coding (e.g. from LeCun’s group)
Learning features from a richer structure of data, e.g., video (learning invariance to out plane rotation)

37. References
Learning Image Representations from Pixel Level via Hierarchical Sparse Coding,
Kai Yu, Yuanqing Lin, John Lafferty. CVPR 2011
Large-scale Image Classification: Fast Feature Extraction and SVM Training,
Yuanqing Lin, Fengjun Lv, Liangliang Cao, Shenghuo Zhu, Ming Yang, Timothee Cour, Thomas Huang, Kai Yu
in CVPR 2011
ECCV 2010 Tutorial, Kai Yu, Andrew Ng (with links to some source codes)
Deep Coding Networks,
Yuanqing Lin, Tong Zhang, Shenghuo Zhu, Kai Yu. In NIPS 2010.
Image Classification using Super-Vector Coding of Local Image Descriptors,
Xi Zhou, Kai Yu, Tong Zhang, and Thomas Huang. In ECCV 2010.
Efficient Highly Over-Complete Sparse Coding using a Mixture Model,
Jianchao Yang, Kai Yu, and Thomas Huang. In ECCV 2010.
Improved Local Coordinate Coding using Local Tangents,
Kai Yu and Tong Zhang. In ICML 2010.
Supervised translation-invariant sparse coding,
Jianchao Yang, Kai Yu, and Thomas Huang, In CVPR 2010
Learning locality-constrained linear coding for image classification,
Jingjun Wang, Jianchao Yang, Kai Yu, Fengjun Lv, Thomas Huang. In CVPR 2010.
Nonlinear learning using local coordinate coding,
Kai Yu, Tong Zhang, and Yihong Gong. In NIPS 2009.
Linear spatial pyramid matching using sparse coding for image classification,
Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. In CVPR 2009.
3/27/2017