1. A Generalized Iterated Shrinkage Algorithm for Non-convex Sparse Coding
Wangmeng Zuo, Deyu Meng, Lei Zhang, Xiangchu Feng, David Zhang
ICCV 2013
Harbin Institute of Technology

2. Overview From L1-norm sparse coding to Lp-norm sparse coding
Existing solvers for Lp-minimization
Generalized shrinkage / thresholding function
Algorithm and analysis
Connections with soft/hard-thresholding functions
Generalized Iterated Shrinkage Algorithms
Experimental results

3. Overcomplete Representation
Compressed Sensing, image restoration, image classification, machine learning, …
Overcomplete Representation
Infinite solutions of x
What’s the optimal?

4. L0-Sparse Coding Impose some prior (constraint) on x: Problems
Sparser is better
Problems
Is the sparsest solution unique?
How can we obtain the optimal solution?

5. Theory: Uniqueness of Sparse Solution (L0)
Nonconvex optimization, intractable
Greedy algorithms: matching pursuit (MP), orthogonal matching pursuit (OMP)

6. Convex Relaxation: L1-Sparse Coding
Problems
When L1- and L0- Sparse Coding have the same solution
Algorithms for L1-Sparse Coding

7. Theory: Uniqueness of Sparse Solution (L1)

8. Theory: Uniqueness of Sparse Solution (L1)
Restricted Isometry Property
Convex, various algorithms have been proposed.

9. Algorithms for L1-Sparse Coding
Iterative shrinkage/thresholding algorithm
Augmented Lagrangian method
Accelerated Proximal Gradient
Homotopy
Primal-Dual Interior-Point Method …
Allen Y. Yang, Zihan Zhou, Arvind Ganesh, Shankar Sastry, and Yi Ma. Fast l1-minimization algorithms for robust face recognition. IEEE Transactions on Image Processing, 2013.
Generalized Iterated Shrinkage Algorithm

10. Lp-norm Approximation
L0-norm: The number of non-zero values
Lp-norm
L1-norm: convex envolope of L0
L0-norm

11. Theory: Uniqueness of Sparse Solution (Lp)
Weaker restricted isometry property is sufficient to guarantee perfect recovery in the Lp case.
R. Chartrand and V. Staneva, "Restricted isometry properties and nonconvex compressive sensing", Inverse Problems, vol. 24, no , pp , 2008

12. Existing Lp-sparse coding algorithms
Analytic solutions: Only suitable for some special cases, e.g., p = 1/2, or p = 1/3.
IRLS, IRL1, ITM_Lp: would not converge to the global optimal solution even for solving the simplest problem
Lookup table
Efficient, pre-computation

13. IRLS for Lp-sparse Coding
(1)
(2)
M. Lai, J. Wang. An unconstrained lq minimization with 0 < q < 1 for sparse solution of under-determined linear systems. SIAM Journal on Optimization, 21(1):82–101, 2011.
Generalized Iterated Shrinkage Algorithm

14. IRL1 for Lp-Sparse Coding
(1)
(2)
E. J. Candes, M. Wakin, S. Boyd. Enhancing sparsity by reweighted l1 minimization. Journal of Fourier Analysis and Applications, 14(5):877–905, 2008.
Generalized Iterated Shrinkage Algorithm

15. ITM_Lp for Lp-Sparse Coding
where
Root of the equation
Y. She. Thresholding-based iterative selection procedures for model selection and shrinkage. Electronic Journal of Statistics, 3:384–415, 2009.
Generalized Iterated Shrinkage Algorithm

16. p = 0.5, λ = 1, and y = 1.3
Generalized Iterated Shrinkage Algorithm

17. Generalized Shrinkage / Thresholding
Keys of soft-thresholding
Thresholding rule: 
Shrinkage rule:
Generalization of soft-thresholding
What’s the thresholding value for Lp?
How to modify the shrinkage rule?

18. (a) y = 1, (b) y = 1.19, (c) y = 1.3, (d) y = 1.5, and (e) y = 1.6
Motivation
(a) y = 1, (b) y = 1.19, (c) y = 1.3, (d) y = 1.5, and (e) y = 1.6

19. Determining the threshold
The first derivative of the nonzero extreme point is zero
The second derivative of the nonzero extreme point higher than zero
The function value at the nonzero extreme point is equivalent with that at zero

20. Determining the shrinkage operator
k = 0, x(k) = |y|
Iterate on k = 0, 1, ..., J
k  k + 1
Generalized Iterated Shrinkage Algorithm

21. Generalized Shrinkage / Thresholding Function
Generalized Iterated Shrinkage Algorithm

22. GST: Theoretical Analysis

23. Connections with soft / hard-thresholding functions
p = 1: GST is equivalent with soft-thresholding
p = 0: GST is equivalent with hard-thresholding

24. Generalized Iterated Shrinkage Algorithms
Lp-sparse coding
Gradient descent
Generalized Shrinkage / Thresholding
Generalized Iterated Shrinkage Algorithm

25. Comparison with Iterated Shrinkage Algorithms
Iterative Shrinkage / Thresholding
Gradient descent
Soft thresholding

26. GISA

27. Sparse gradient based image deconvolution
Generalized Iterated Shrinkage Algorithm

28. Application I: Deconvolution

29. Application I: Deconvolution

30. Application II: Face Recognition
Extended YaleB

32. Conclusion
Compared with the state-of-the-art methods, GISA is theoretically solid, easy to understand and efficient to implement, and it can converge to a more accurate solution.
Compared with LUT, GISA is more general and does not need to compute and store the look-up tables.
GISA can be readily used to solve the many lp–norm minimization problems in various vision and learning applications.
Generalized Iterated Shrinkage Algorithm

33. Looking forward Applications to other vision problems.
Incorporation of the primal-dual algorithm for better solution
Extension of GISA for constrained Lp-minimization, e.g.,