1. NESTA: A Fast and Accurate First-Order Method for Sparse Recovery
Reading Group
NESTA: A Fast and Accurate First-Order Method for Sparse Recovery
(STEPHEN BECKER, JEROME BOBIN and EMMANUEL J. CANDES)
Presenter: Feng Lin
Cognitive Radio Institute
ECE / CMR
Tennessee Technological University
March 18, 2011

2. Outline Introduction Nesterov’s method NESTA algorithm details
Accelerate NESTA with continuation
Accurate optimization
Comparison among other algorithms
Problem solving applications
Conclusion
11/17/2018

3. Motivation Solving large-scale problems is challenging
Standard second-order methods are accurate but problematic to compute lots of newton steps.
A lot of first order methods may be faster but not accurate.
So we would like to pursue the high accuracy with more digits of precision
Another motivation is solving signal is not exactly sparse, but rather approximately sparse, as in real world compressible signals.
11/17/2018

4. Advantage of NESTA
Ideally suited for solving large-scale compressed sensing reconstruction problems
1) It is computationally efficient
2) It is accurate and returns solutions with several correct digits
3) It is flexible and amenable to many kinds of reconstruction problems
4) It is robust in the sense that its excellent performance across a wide range of problems does not depend on the fine tuning of several parameters
11/17/2018

5. Introduction
Signal model of Compressed sensing
11/17/2018

6. Introduction
Some optimization problems that will be discussed throughout
Quadratically constrained l1-minimization problem
Equivalent Lagrangian form
Lasso
11/17/2018

7. Nesterov’s method Minimizing smooth convex functions
Function f is assumed to be differentiable and its gradient is Lipschitz and obeys
L is an upper bound on Lipschitz constant
Nesterov's algorithm minimizes f(x) over Qp(primal feasible set) by iteratively estimating three sequences xk, yk and zk while smoothing the feasible set Qp.
11/17/2018

8. Nesterov’s method Minimizing nonsmooth convex functions. Recast as
Nesterov proposed smooth method
is Lipschitz with constant
11/17/2018

9. Yk is the current guess of the optimal solution.
Zk involves a weighted sum of already computed gradients
from theoretical analysis of other papers.
11/17/2018

10. Nesterov’s method :
Convergence rate
11/17/2018

11. NESTA The algorithm uses two ideas due to Yurii Nesterov.
The first idea is an accelerated convergence scheme for first-order methods, giving the optimal convergence rate for this class of problems.
The second idea is a smoothing technique that replaces the non-smooth l1 norm with a smooth version.
We assume that A*A is an orthogonal project, i.e. the rows of A are orthogonal.
11/17/2018

12. NESTA Target: we wish to solve the BP problem Form smooth function:
11/17/2018

13. Lagrangian for this problem is
Updating yk:
Where
Lagrangian for this problem is
We can get the KKT conditions
11/17/2018

14. Solve the KKT function, we get
Updating zk
We may choose
As before, we can get
11/17/2018

15. Parameters selection A single smoothing parameter µ
A suitable stopping criterion
Terminate the algorithm when the relative variation of fµ is small, i.e , where , depending upon the desired accuracy.
11/17/2018

16. Accelerate NESTA with continuation
Continuation is a very useful tool to increase the speed of convergence, in particular when dealing with large-scale problems and high dynamic range signals.
For NESTA algorithm, the convergence rate obeys
Since Lµ is proportional to 1/µ, the larger µ and the faster the convergence.
Choosing a good guess x0 close to xµ also improving the rate of convergence, because low value of
11/17/2018

17. Interpretation
11/17/2018

18. 11/17/2018

19. Continuation step T = 4 , 5 ,6 leads to reasonable results.
11/17/2018

20. Accurate optimization
Two criteria
Relative error on the objective function
Accuracy of the optimal solution itself
FISTA NESTA comparison
11/17/2018

21. µ and accuracy When µ decreases,
the accuracy of NESTA increases, decreasing µ by a factor of 10 gives about 1 additional digit of accuracy on the optimal value.
NA increase accordingly.
µ = 0.02 seems a reasonable choice to balance these two considerations
11/17/2018

22. Comparison among other algorithms
NESTA
Gradient Projections for Sparse Reconstruction (GPSR)
Sparse reconstruction by separable approximation (SpaRSA)
l1 regularized least squares (l1_ls)
Spectral projected gradient (SPGL1)
Fixed Point Continuation method (FPC)
FPC Active Set (FPC-AS)
Bregman
Fast Iterative Soft-Thresholding Algorithm (FISTA)
11/17/2018

23. Constrained versus unconstrained minimization
NESTA solve the constrained problem (BP)
SPGL1 solve the constrained problem (LST)
while all other methods tested solve the unconstrained problem (QP).
In general, solving an unconstrained problem is much easier than a constrained problem.
11/17/2018

24. 11/17/2018

25. 11/17/2018

26. Application : Nonstandard sparse reconstruction: l1 analysis
For real-world signals, the signal is not sparse, but it may be sparse (or be approximately-sparse, i.e. most energy is concentrated in only a few coefficients) in a dictionary W, can be represented as X=Wα.
Traditionally, we attempts reconstruction by solving synthesis-based approach, simply treat (AW) as a single operator.
If W is orthonormal, it is equivalent to
We can apply NESTA, only need to change the step1, while step 2, 3 remains
11/17/2018

27. Application : Total-variation minimization
The TV norm of a 2D digital object x[i; j] is given
where D1 and D2 are the horizontal and vertical differences
We also can form
The rest are the same
11/17/2018

28. Extension
This paper focused on the situation in which A*A is a projector (the rows of A are orthonormal).
Most computationally friendly compressed sensing are of this form
It allows fast computations of the two sequence of iterates yk, zk.
However, NESTA can cope with the problem with A*A is not a projection (or not diagonal).
We need to solve
Then
A good rule for selecting reasonable λ is critical for extending NESTA to a general problem
11/17/2018

29. Conclusions
NESTA , an iterative algorithm with fast convergence rate based on Nesterov’s algorithm, is a method of choice for solving large-scale problems.
NESTA is accurate that can find the first 4 or 5 significant digits of the optimal solution.
NESTA can be adapted to solve many problems beyond l1 minimization with the same efficiency, such as l1 analysis, total-variation (TV) minimization problems.
11/17/2018

30. Thank you!
11/17/2018