1. Online Learning for Matrix Factorization and Sparse Coding
Julien Mairal, Francis Bach, Jean Ponce and Guillermo Sapiro
Journal of Machine Learning Research 2010

2. Introduction
This paper focuses on the large scale matrix factorization problem, including
Dictionary learning for sparse coding
Non-negative matrix factorization (NMF)
Sparse principal component analysis (SPCA)
Contributions of this paper:
An iterative online algorithm is proposed for large scale matrix factorization
This algorithm is proved to converge almost surely to a stationary point of the objective function
This algorithm is shown to be much faster than previous methods in the experiment.

3. Problem Statement Classical dictionary learning problem
Given a finite training set , the objective is to optimize the following function
where
Online Learning
This algorithm process one sample (or a mini-batch) at a time and sequentially minimize the following function:

4. Basic Algorithm

5. Dictionary Update

6. Optimizing the Algorithm
Handling fixed-sized data sets
Scaling the “past” data
Mini-batch extension

7. Proof of Convergence
Assumptions:
Main results

8. Extensions to Matrix Factorization
Non-negative matrix factorization (NMF)
Non-negative sparse coding (NNSC)
Sparse principal component analysis (SPCA)

9. Data for Experiment
1.25 million patches from Pascal VOC’06 image database

10. Online VS. Batch
Training data size: 1 million
OL1:
OL2:
OL3:

11. Comparison with NMF and NNSC

12. Face Results

13. Image Patches Results

14. Inpainting Results Image size: 12-Megapixel
Dictionary with 256 elements
Training data: 7 million 12 by 12 color patches

15. Conclusion
A new online algorithm for learning dictionaries adapted to sparse coding tasks, and proven its convergence.
Experiments demonstrate that this algorithm is significantly faster than existing batch methods.
This algorithm can be extended to other matrix factorization problems such as non-negative matrix factorization and sparse principal component analysis.