A Generalized Iterated Shrinkage Algorithm for Non-convex Sparse Coding

1
A Generalized Iterated Shrinkage Algorithm for
Non-convex Sparse Coding

• Wangmeng Zuo, Deyu Meng, Lei Zhang, Xiangchu
  Feng, David Zhang
• ICCV 2013
• wmzuo_at_hit.edu.cn
• Harbin Institute of Technology

2
Overview

• From L1-norm sparse coding to Lp-norm sparse
  coding
• Existing solvers for Lp-minimization
• Generalized shrinkage / thresholding function
• Algorithm and analysis
• Connections with soft/hard-thresholding functions
• Generalized Iterated Shrinkage Algorithms
• Experimental results

3
Overcomplete Representation

• Compressed Sensing, image restoration, image
  classification, machine learning,
• Overcomplete Representation
• Infinite solutions of x
• Whats the optimal?

4
L0-Sparse Coding

• Impose some prior (constraint) on x
• Sparser is better
• Problems
• Is the sparsest solution unique?
• How can we obtain the optimal solution?

5
Theory Uniqueness of Sparse Solution (L0)

• Nonconvex optimization, intractable
• Greedy algorithms matching pursuit (MP),
  orthogonal matching pursuit (OMP)

6
Convex Relaxation L1-Sparse Coding

• L1-Sparse Coding
• Problems
• When L1- and L0- Sparse Coding have the same
  solution
• Algorithms for L1-Sparse Coding

7
Theory Uniqueness of Sparse Solution (L1)
8
Theory Uniqueness of Sparse Solution (L1)

• Restricted Isometry Property

• Convex, various algorithms have been proposed.

9
Algorithms for L1-Sparse Coding

• Iterative shrinkage/thresholding algorithm
• Augmented Lagrangian method
• Accelerated Proximal Gradient
• Homotopy
• Primal-Dual Interior-Point Method

Allen Y. Yang, Zihan Zhou, Arvind Ganesh, Shankar
Sastry, and Yi Ma. Fast l1-minimization
algorithms for robust face recognition. IEEE
Transactions on Image Processing, 2013.
10
Lp-norm Approximation

• L0-norm The number of non-zero values
• Lp-norm
• L1-norm convex envolope of L0
• L0-norm

11
Theory Uniqueness of Sparse Solution (Lp)

• Weaker restricted isometry property is sufficient
  to guarantee perfect recovery in the Lp case.

R. Chartrand and V. Staneva, "Restricted isometry
properties and nonconvex compressive sensing",
Inverse Problems, vol. 24, no. 035020, pp. 1--14,
2008
12
Existing Lp-sparse coding algorithms

• Analytic solutions Only suitable for some
  special cases, e.g., p 1/2, or p 1/3.
• IRLS, IRL1, ITM_Lp would not converge to the
  global optimal solution even for solving the
  simplest problem
• Lookup table
• Efficient, pre-computation

13
IRLS for Lp-sparse Coding

• IRLS
• (1)
• (2)

 
 
M. Lai, J. Wang. An unconstrained lq minimization
with 0 lt q lt 1 for sparse solution of
under-determined linear systems. SIAM Journal on
Optimization, 21(1)82101, 2011.
14
IRL1 for Lp-Sparse Coding

• IRL1
• (1)
• (2)

E. J. Candes, M. Wakin, S. Boyd. Enhancing
sparsity by reweighted l1 minimization. Journal
of Fourier Analysis and Applications,
14(5)877905, 2008.
15
ITM_Lp for Lp-Sparse Coding

• ITM_Lp
• where

Root of the equation
 
 
Y. She. Thresholding-based iterative selection
procedures for model selection and shrinkage.
Electronic Journal of Statistics, 3384415, 2009.
16
p 0.5, ? 1, and y 1.3
17
Generalized Shrinkage / Thresholding

• Keys of soft-thresholding
• Thresholding rule ?
• Shrinkage rule
• Generalization of soft-thresholding
• Whats the thresholding value for Lp?
• How to modify the shrinkage rule?

18
Motivation
 

• (a) y 1, (b) y 1.19, (c) y 1.3, (d) y
  1.5, and (e) y 1.6

19
Determining the threshold

• The first derivative of the nonzero extreme point
  is zero
• The second derivative of the nonzero extreme
  point higher than zero
• The function value at the nonzero extreme point
  is equivalent with that at zero

20
Determining the shrinkage operator

• k 0, x(k) y
• Iterate on k 0, 1, ..., J
• k ? k 1

 
21
Generalized Shrinkage / Thresholding Function
22
GST Theoretical Analysis
23
Connections with soft / hard-thresholding
functions

• p 1 GST is equivalent with soft-thresholding
• p 0 GST is equivalent with hard-thresholding

24
Generalized Iterated Shrinkage Algorithms

• Lp-sparse coding
• Gradient descent
• Generalized Shrinkage / Thresholding

25
Comparison with Iterated Shrinkage Algorithms

• Iterative Shrinkage / Thresholding
• Gradient descent
• Soft thresholding

26
GISA
27
Sparse gradient based image deconvolution
28
Application I Deconvolution
29
Application I Deconvolution
30
Application II Face Recognition

• Extended YaleB

31
(No Transcript)
32
Conclusion

• Compared with the state-of-the-art methods, GISA
  is theoretically solid, easy to understand and
  efficient to implement, and it can converge to a
  more accurate solution.
• Compared with LUT, GISA is more general and does
  not need to compute and store the look-up
  tables.
• GISA can be readily used to solve the many
  lpnorm minimization problems in various vision
  and learning applications.

33
Looking forward

• Applications to other vision problems.
• Incorporation of the primal-dual algorithm for
  better solution
• Extension of GISA for constrained
  Lp-minimization, e.g.,