1. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter Sparse & Redundant Representation Modeling of Images OMP versus Batch OMP Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000, Israel These slides were prepared by Matan Protter and used for a summer school on sparse approximation in PCMI (Park City Mathematical Institute (2010)

2. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 2 Recall: The Thresholding Alg. Initialization 1. 2. Main Iteration 1. 2. 3. Stop Yes No Thresholding finds the set of atoms approximating the solution of

3. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 3 Recall: The OMP Algorithm Initialization Main Iteration 1. 2. 3. 4. 5. Stop Yes No OMP finds one atom at a time for approximating the solution of

4. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 4 Few Notes Stages 1 & 2 in loop: To find the next atom to use: –Compute the projection of the residual onto the dictionary –Select the atom with the largest (magnitude) projection In Stage 4: –One can use Matlab’s “\” operator to solve the Least- Squares problem, or –Even exploit the “recursive option” of Least-Squares with growing number of unknowns.

5. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 5 Can we Speed Up the OMP?

6. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 6 Observation 1 In step 1, we compute for each atom in the dictionary Suppose during the loop, we have found support S and coefficient We can write:

7. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 7 Observation 1 – Cont. If we work on one signal – this doesn’t help If we work on many signals (>> number of atoms), we can compute ONCE in advance (this is actually the multiplication ) We also need to compute once for each signal

8. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 8 Observation 1 – Cont. # of operations needed for computing –Regular OMP n – dimension of the signal –Modified Version |S| - the size of the current support (assuming there are enough signals that computing is negligible) We term this version “Batch-OMP”

9. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 9 Observation 1 - Practicalities Pre-compute Denote as the sub-matrix of obtained by taking only the columns belonging in S

10. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 10 Observation 1 - Summary Pre-compute and To find the atom with largest inner product with the residual, compute: The vector of the coefficients belonging to the support

11. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 11 Implications Initialization Main Iteration 1. 2. 3. 4. 5. Stop Yes No OMP finds one atom at a time for approximating the solution of Replaced Removed

12. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 12 Observation 2 Assuming we have an error threshold as condition, we need to compute the norm of the residual in each stage In observation 1, we avoid computing the residual altogether … Can we still avoid it? Answer: Yes, this is possible …

13. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 13 Observation 2 – Cont. The norm of the residual is cheap to compute using the norm from the previous iteration Where: The coefficients of the atoms in the support in iteration n A sub-matrix of by taking the rows and columns in the support in the n-th iteration S n

14. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 14 Observation 2 – Cont. Denote Then: Therefore: compute in each iteration, and update the norm of the residual accordingly

15. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 15 Observation 2 – Cont. Is it cheap to compute ? Yes! is already computed for the update of and then, is simply a dot product between two short vectors of length |S n | each

16. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 16 Denote the norm of the residual in iteration n by Initialize to At each stage, compute And update the norm of the residual Observation 2 – Cont.

17. Sparse & Redundant Representation Modeling of Images Problem Solving Session 1: Greedy Pursuit Algorithms By: Matan Protter 17 Batch-OMP Summary Start with the regular OMP algorithm Add pre-processing steps: –Compute Modify the atom selection stage using Modify the residual update stage as shown before – ONLY if you intend to stop by the residual norm