Efficient Sparse Coding Algorithms Liang Sun Sun.Liang@asu.edu Arizona State University

Paper Source Honglak Lee, Alexis Battle, Rajat Raina, Andrew Y. Ng. Efficient Sparse Coding Algorithms. NIPS 2007.

Main Contributions It proposes a new efficient algorithm to solve LASSO A two stage optimization algorithm is proposed to the coding problem

Sparse Coding Problem Sparse coding is a method for discovering good basis vectors automatically using only unlabeled data It is similar to PCA Given a training set of m vectors where , we attempt to find a succinct representation for each xi using basis vectors and a sparse vector such that Note that the basis can be overcomplete, i.e., n>k

The basis act as the principal components in PCA, and they capture a large number of patterns in the input data The optimization problem in sparse coding where and is a sparse penalty function

A New Algorithm to Solve LASSO The formulation of LASSO where x, y are vectors and A is a matrix Basic idea of the new algorithm The difficulty of this problem lies in We guess the sign of each component of x

Feature-sign Search Algorithm

Proof of Feature-sign Search Algorithm – (I)

Proof of Feature-sign Search Algorithm – (II)

Proof of Feature-sign Search Algorithm – (III)

Use Dual to simplify computation Original Problem

Experiment – (I) The comparison of Feature-sign search algorithm and other algorithms for LASSO

Experiment – (II) The comparison of the two-stage algorithm and other algorithms