Efficient Sparse Coding Algorithms

1
Efficient Sparse Coding Algorithms

• Liang Sun
• Sun.Liang_at_asu.edu
• Arizona State University

2
Paper Source

• Honglak Lee, Alexis Battle, Rajat Raina, Andrew
  Y. Ng. Efficient Sparse Coding Algorithms. NIPS
  2007.

3
Main Contributions

• It proposes a new efficient algorithm to solve
  LASSO
• A two stage optimization algorithm is proposed to
  the coding problem

4
Sparse Coding Problem

• Sparse coding is a method for discovering good
  basis vectors automatically using only unlabeled
  data
• It is similar to PCA
• Given a training set of m vectors
  where , we attempt to find a
  succinct representation for each xi using basis
  vectors
• and a sparse vector
  such that
• Note that the basis can be overcomplete, i.e., ngtk

5

• The basis act as the principal components in PCA,
  and they capture a large number of patterns in
  the input data
• The optimization problem in sparse coding
• where
• and is a sparse penalty function

6
A New Algorithm to Solve LASSO

• The formulation of LASSO
• where x, y are vectors and A is a matrix
• Basic idea of the new algorithm
• The difficulty of this problem lies in
• We guess the sign of each component of x

7
Feature-sign Search Algorithm
8
Proof of Feature-sign Search Algorithm (I)
9
Proof of Feature-sign Search Algorithm (II)
10
Proof of Feature-sign Search Algorithm (III)
11
Use Dual to simplify computation

• Original Problem

12
Experiment (I)

• The comparison of Feature-sign search algorithm
  and other algorithms for LASSO

13
Experiment (II)

• The comparison of the two-stage algorithm and
  other algorithms