+ Quadratic Programming and Duality Sivaraman Balakrishnan
+ Outline Quadratic Programs General Lagrangian Duality Lagrangian Duality in QPs
+ Norm approximation Problem Interpretation Geometric – try to find projection of b into ran(A) Statistical – try to find solution to b = Ax + v v is a measurement noise (choose norm so that v is small in that norm) Several others
+ Examples -- Least Squares Regression -- Chebyshev -- Least Median Regression More generally can use *any* convex penalty function
+ Picture from BV
+ Least norm Perfect measurements Not enough of them Heart of something known as compressed sensing Related to regularized regression in the noisy case
+ Smooth signal reconstruction S(x) is a smoothness penalty Least squares penalty Smooths out noise and sharp transitions Total variation (peak to valley intuition) Smooths out noise but preserves sharp transitions
+ Euclidean Projection Very fundamental idea in constrained minimization Efficient algorithms to project onto many many convex sets (norm balls, special polyhedra etc) More generally finding minimum distance between polyhedra is a QP
+ Quadratic Programming Duality
+ General recipe Form Lagrangian How to figure out signs?
+ Primal & Dual Functions Primal Dual
+ Primal & Dual Programs Primal Programs Constraints are now implicit in the primal Dual Program
+ Lagrangian Properties Can extract primal and dual problem Dual problem is always concave Proof Dual problem is always a lower bound on primal Proof Strong duality gives complementary slackness Proof
+ Some examples of QP duality Consider the example from class Lets try to derive dual using Lagrangian
+ General PSD QP Primal Dual
+ SVM – Lagrange Dual Primal SVM Dual SVM Recovering Primal Variables and Complementary Slackness