1. Efficient Sparse Coding Algorithms
Liang Sun
Arizona State University

2. Paper Source
Honglak Lee, Alexis Battle, Rajat Raina, Andrew Y. Ng. Efficient Sparse Coding Algorithms. NIPS 2007.

3. Main Contributions
It proposes a new efficient algorithm to solve LASSO
A two stage optimization algorithm is proposed to the coding problem

4. Sparse Coding Problem
Sparse coding is a method for discovering good basis vectors automatically using only unlabeled data
It is similar to PCA
Given a training set of m vectors where , we attempt to find a succinct representation for each xi using basis vectors
and a sparse vector such that
Note that the basis can be overcomplete, i.e., n>k

5. The basis act as the principal components in PCA, and they capture a large number of patterns in the input data
The optimization problem in sparse coding
where
and is a sparse penalty function

6. A New Algorithm to Solve LASSO
The formulation of LASSO
where x, y are vectors and A is a matrix
Basic idea of the new algorithm
The difficulty of this problem lies in
We guess the sign of each component of x

7. Feature-sign Search Algorithm

8. Proof of Feature-sign Search Algorithm – (I)

9. Proof of Feature-sign Search Algorithm – (II)

10. Proof of Feature-sign Search Algorithm – (III)

11. Use Dual to simplify computation
Original Problem

12. Experiment – (I)
The comparison of Feature-sign search algorithm and other algorithms for LASSO

13. Experiment – (II)
The comparison of the two-stage algorithm and other algorithms