Locality-constrained Linear Coding for Image Classification Presenter : Han-Mu Park

Contents Introduction Coding methods Proposed method Locality-constrained Linear Coding for Image Classification, CVPR 2010 Contents Introduction Coding methods Proposed method Experimental results Conclusion References

Introduction Bag-of-Words (BoW) model Locality-constrained Linear Coding for Image Classification, CVPR 2010 Introduction Bag-of-Words (BoW) model An image is represented as a collection of visual words. Generally, to represent the collection, histogram of words form is used.

Introduction General Spatial Pyramid Matching frameworks Locality-constrained Linear Coding for Image Classification, CVPR 2010 Introduction General Spatial Pyramid Matching frameworks Feature extraction SIFT HOG etc Coding Vector Quantization Sparse coding Pooling Max pooling Sum pooling Spatial Pyramid Matching framework [J.Wang2010]

Introduction General Spatial Pyramid Matching frameworks Locality-constrained Linear Coding for Image Classification, CVPR 2010 Introduction General Spatial Pyramid Matching frameworks Spatial Pyramid Matching framework [J.Wang2010]

Coding methods Coding methods Vector quantization (VQ) Locality-constrained Linear Coding for Image Classification, CVPR 2010 Coding methods Coding methods Vector quantization (VQ) Sparce coding (SC) Locality-constrained Linear Coding (LLC) [J.Wang2010]

Coding methods Vector quantization (VQ) Hard quantization method Locality-constrained Linear Coding for Image Classification, CVPR 2010 Coding methods Vector quantization (VQ) Hard quantization method A set of 𝐷-dimensional local descriptors 𝑋= 𝑥 1 , 𝑥 2 , …,𝑥 𝑁 ∈ 𝑅 𝐷×𝑁 Codebook with 𝑀 entries 𝐵= 𝑏 1 , 𝑏 2 ,…, 𝑏 𝑀 ∈ 𝑅 𝐷×𝑀 Objective function 𝑎𝑟𝑔 min 𝐶 𝑖=1 𝑁 𝑥 𝑖 −𝐵 𝑐 𝑖 2 𝑠.𝑡. 𝑐 𝑖 𝑙 0 =1, 𝑐 𝑖 𝑙 1 =1, 𝑐 𝑖 ≽0, ∀𝑖 Where 𝐶= 𝑐 1 , 𝑐 2 ,…, 𝑐 𝑁 is the set of codes for X

Coding methods Sparse coding (SC) Soft quantization method Locality-constrained Linear Coding for Image Classification, CVPR 2010 Coding methods Sparse coding (SC) Soft quantization method Relaxed the cardinality constraint Objective function 𝑎𝑟𝑔 min 𝐶 𝑖=1 𝑁 𝑥 𝑖 −𝐵 𝑐 𝑖 2 +𝜆 𝑐 𝑖 𝑙 1 The roles of sparsity regularization term Because the codebook 𝐵 is usually over-complete 𝑀>𝐷 , it is necessary to ensure that the under-determined system has a unique solution. Sparsity allows the learned representation to capture salient patterns of local descriptors. The sparse coding can achieve much less quantization error than VQ.

Proposed method Locality-constrained Linear Coding (LLC) Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method Locality-constrained Linear Coding (LLC) Replaced the sparsity regularization term with new constraint. Objective function 𝑎𝑟𝑔 min 𝐶 𝑖=1 𝑁 𝑥 𝑖 −𝐵 𝑐 𝑖 2 + 𝜆 𝑑 𝑖 ⊙ 𝑐 𝑖 2 𝑠.𝑡. 1 𝑇 𝑐 𝑖 =1, ∀𝑖 ⊙ : the element-wise multiplication 𝑑 𝑖 ∈ 𝑅 𝑀 𝑑 𝑖 = exp 𝑑𝑖𝑠𝑡 𝑥 𝑖 ,𝐵 𝜎 Where 𝑑𝑖𝑠𝑡 𝑥 𝑖 ,𝐵 = 𝑑𝑖𝑠𝑡 𝑥 𝑖 , 𝑏 1 ,𝑑𝑖𝑠𝑡 𝑥 𝑖 , 𝑏 2 ,…,𝑑𝑖𝑠𝑡 𝑥 𝑖 , 𝑏 𝑀 𝑇

Proposed method Properties of LLC Better reconstruction Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method Properties of LLC Better reconstruction Because LLC represents each descriptor by using multiple weighted bases (codewords), it has less reconstruction error than VQ. Local smooth sparsity Because the regularization term of 𝑙 1 𝑛𝑜𝑟𝑚 in SC is not smooth, therefore, SC loses correlations between codes. Analytical solution The solution of LLC can be derived analytically by 𝑐 𝑖 = 1 𝐶 𝑖 +𝜆𝑑𝑖𝑎𝑔 𝑑 𝑐 𝑖 = 𝑐 𝑖 1 𝑇 𝑐 𝑖 Where 𝐶 𝑖 = 𝐵−1 𝑥 𝑖 𝑇 𝐵−1 𝑥 𝑖 𝑇 𝑇

Proposed method Approximated LLC for fast encoding Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method Approximated LLC for fast encoding The LLC selects the local bases for each descriptor to form a local coordinate system. To speedup the encoding process, authors used 𝐾 (𝐾<𝐷<𝑀) nearest neighbors of 𝑥 𝑖 as the local bases 𝐵 𝑖 , and solve a much smaller linear system to get the codes min 𝐶 𝑖=1 𝑁 𝑥 𝑖 − 𝑐 𝑖 𝐵 𝑖 2 𝑠.𝑡. 1 𝑇 𝑐 𝑖 =1, ∀𝑖 The reduced computation complexity 𝑂 𝑀 2 ⟶𝑂 𝑀+ 𝐾 2 , where 𝐾≪𝑀

Proposed method Codebook optimization Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method Codebook optimization To improve the accuracy, authors trained the codebook to optimize for LLC codes. The optimal codebook 𝐵 ∗ can be obtained by 𝑎𝑟𝑔 min 𝐶,𝐵 𝑖=1 𝑁 𝑥 𝑖 −𝐵 𝑐 𝑖 2 +𝜆 𝑑 𝑖 ⊙ 𝑐 𝑖 2 𝑠.𝑡. 1 𝑇 𝑐 𝑖 =1, ∀𝑖 𝑏 𝑗 2 ≤1, ∀𝑗 This can be solved by using Coordinate Descent method. However, because the number of training descriptors 𝑁 is usually very large, the huge memory space is needed to solve that problem.

Proposed method Incremental codebook optimization Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method Incremental codebook optimization First, initialize 𝐵 by using K-means clustering. Then loop through all the training descriptors to update 𝐵 incrementally. In each iteration, we take in a single (or a small set of) examples 𝑥 𝑖 , and solve original objective function to obtain the corresponding LLC codes. [J.Wang2010]

Proposed method Incremental codebook optimization Locality-constrained Linear Coding for Image Classification, CVPR 2010 Proposed method Incremental codebook optimization Then select bases 𝐵 𝑖 whose corresponding weights are larger than predefined threshold, and refit 𝑥 𝑖 without the locality constraint. The obtained code is used to update the basis in a gradient descent fashion. [J.Wang2010]

Experimental results Performance of codebook Locality-constrained Linear Coding for Image Classification, CVPR 2010 Experimental results Performance of codebook

Experimental results Performance under different neighbors Locality-constrained Linear Coding for Image Classification, CVPR 2010 Experimental results Performance under different neighbors

Experimental results Results using Pascal VOC 2007 Locality-constrained Linear Coding for Image Classification, CVPR 2010 Experimental results Results using Pascal VOC 2007

Conclusion Contribution Locality-constrained Linear Coding for Image Classification, CVPR 2010 Conclusion Contribution In this paper, the Locality-constrained Linear Coding (LCC) method is proposed Better reconstruction Local smooth sparsity Analytical solution For speedup, K-nearest neighbors algorithm is used. To optimize the accuracy, incremental codebook learning is proposed for LCC.

Locality-constrained Linear Coding for Image Classification, CVPR 2010 References [1] J. Wang, J. Yang, K. Yu, F. Lv, T. Huang, Y. Gong, “Locality-constrained Linear Coding for Image Classification,” CVPR 2010.

Thank you!