1. + Quadratic Programming and Duality Sivaraman Balakrishnan

2. + Outline Quadratic Programs General Lagrangian Duality Lagrangian Duality in QPs

3. + Norm approximation Problem Interpretation Geometric – try to find projection of b into ran(A) Statistical – try to find solution to b = Ax + v v is a measurement noise (choose norm so that v is small in that norm) Several others

4. + Examples -- Least Squares Regression -- Chebyshev -- Least Median Regression More generally can use *any* convex penalty function

5. + Picture from BV

6. + Least norm Perfect measurements Not enough of them  Heart of something known as compressed sensing Related to regularized regression in the noisy case

7. + Smooth signal reconstruction S(x) is a smoothness penalty Least squares penalty Smooths out noise and sharp transitions Total variation (peak to valley intuition) Smooths out noise but preserves sharp transitions

8. + Euclidean Projection Very fundamental idea in constrained minimization Efficient algorithms to project onto many many convex sets (norm balls, special polyhedra etc) More generally finding minimum distance between polyhedra is a QP

9. + Quadratic Programming Duality

10. + General recipe Form Lagrangian How to figure out signs?

11. + Primal & Dual Functions Primal Dual

12. + Primal & Dual Programs Primal Programs Constraints are now implicit in the primal Dual Program

13. + Lagrangian Properties Can extract primal and dual problem Dual problem is always concave Proof Dual problem is always a lower bound on primal Proof Strong duality gives complementary slackness Proof

14. + Some examples of QP duality Consider the example from class Lets try to derive dual using Lagrangian

15. + General PSD QP Primal Dual

16. + SVM – Lagrange Dual Primal SVM Dual SVM Recovering Primal Variables and Complementary Slackness