1. C&O 355 Lecture 15 N. Harvey TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AA A A A A A A A A

2. Topics Subgradient Inequality Characterizations of Convex Functions Convex Minimization over a Polyhedron (Mini)-KKT Theorem Smallest Enclosing Ball Problem

3. Subgradient Inequality Prop: Suppose f : R ! R is differentiable. Then f is convex iff f(y) ¸ f(x) + f(x)(y-x) 8 x,y 2 R Proof: ( : See Notes Section 3.2. ) : Exercise for Assignment 4. ¤

4. Convexity and Second Derivative Prop: Suppose f : R ! R is twice-differentiable. Then f is convex iff f(x) ¸ 0 8 x 2 R. Proof: See Notes Section 3.2.

5. Subgradient Inequality in R n Prop: Suppose f : R n ! R is differentiable. Then f is convex iff f(y) ¸ f(x) + r f (x) T (y-x) 8 x,y 2 R Proof: ( : Exercise for Assignment 4. ) : See Notes Section 3.2. ¤

6. Minimizing over a Convex Set Prop: Let C µ R n be a convex set. Let f : R n ! R be convex and differentiable. Then x minimizes f over C iff r f(x) T (z-x) ¸ 0 8 z 2 C. Proof: ( direction Direct from subgradient inequality. f(z) ¸ f(x) + r f(x) T (z-x) ¸ f(x) Subgradient inequality Our hypothesis

7. Minimizing over a Convex Set Prop: Let C µ R n be a convex set. Let f : R n ! R be convex and differentiable. Then x minimizes f over C iff r f(x) T (z-x) ¸ 0 8 z 2 C. Proof: ) direction Let x be a minimizer, let z 2 C and let y = z-x. Recall that r f(x) T y = f(x;y) = lim t ! 0 f(x+ty)-f(x). If limit is negative then we have f(x+ty)<f(x) for some t 2 [0,1], contradicting that x is a minimizer. So the limit is non-negative, and r f(x) T y ¸ 0. ¥ t

8. Positive Semidefinite Matrices (again) Assume M is symmetric Old definition: M is PSD if 9 V s.t. M = V T V. New definition: M is PSD if y T My ¸ 0 8 y 2 R n. Claim: Old ) New. Proof: y T My = y T V T Vy = k Vy k 2 ¸ 0. Claim: New ) Old. Proof: Based on spectral decomposition of M.

9. Convexity and Hessian Prop: Let f: R n ! R be a C 2 -function. Let H(x) denote the Hessian of f at point x. Then f is convex iff H(x) is PSD 8 x 2 R n. Proof: ) direction Fix y 2 R n. Consider function g y ( ® ) = f(x+ ® y). Convexity of g y follows from convexity of f. Thus g y (0) ¸ 0. (convexity & 2 nd derivative) Fact: g y (0) = y T H(x) y (stated in Lecture 14) So y T H(x) y ¸ 0 8 y 2 R n ) H(x) is PSD.

10. Convexity and Hessian Prop: Let f: R n ! R be a C 2 -function. Let H(x) denote the Hessian of f at point x. Then f is convex iff H(x) is PSD 8 x 2 R n. Proof: ( direction. Fix x,y 2 R n. Define g : [0,1] ! R by g( ® ) = f(x+ ® (y-x)). For all ®, g( ® ) =(y-x) T H(x+ ® (y-x)) (y-x) ¸ 0. (The equality was stated in Lecture 14. The inequality holds since H is PSD.) So g is convex. (By Prop Convexity and Second Derivative) f((1- ® )x+ ® y) = g( ® ) · (1- ® )g(0) + ® g(1) = (1- ® )f(x) + ® f(y). ¥

11. Hessian Example Example: Let M be a symmetric n x n matrix. Let z 2 R n. Define f : R n ! R by f(x) = x T Mx - x T z. Note f(x) = § i § j M i,j x i x j - § i x i z i. Taking partial derivatives r f(x) i = 2 § j M i,j x j - z i. So r f(x) = 2Mx-z. Recall the Hessian at x is H(x) = r ( r f(x)). So H(x) = 2M.

12. Smallest Ball Problem Let {p 1,…,p n } be points in R d. Find (unique!) ball of smallest volume (not an ellipsoid!) that contains all the p i s. In other words, we want to solve: min { r : 9 y 2 R d s.t. p i 2 B(y,r) 8 i }

13. Smallest Ball Problem Let {p 1,…,p n } be points in R d. Find (unique!) ball of smallest volume (not an ellipsoid!) that contains all the p i s. In other words, we want to solve: min { r : 9 y 2 R d s.t. p i 2 B(y,r) 8 i } We will formulate this as a convex program. In fact, our convex program will be of the form min { f(x) : Ax=b, x ¸ 0 }, where f is convex. Minimizing a convex function over an (equality form) polyhedron To solve this, we will need optimality conditions for convex programs.

14. (Mini)-KKT Theorem j th column of A Theorem: Let f: R n ! R be a convex, C 2 function. Let x 2 R n be a feasible solution to the convex program min { f(x) : Ax=b, x ¸ 0 } Then x is optimal iff 9 y 2 R m s.t. 1) r f(x) T + y T A ¸ 0, 2) For all j, if x j >0 then r f(x) j + y T A j = 0. Proven by Karush in 1939 (his Masters thesis!), and by Kuhn and Tucker in 1951.

15. (Mini)-KKT Theorem Theorem: Let f: R n ! R be a convex, C 2 function. Let x 2 R n be a feasible solution to the convex program min { f(x) : Ax=b, x ¸ 0 } Then x is optimal iff 9 y 2 R m s.t. 1) r f(x) T + y T A ¸ 0, 2) For all j, if x j >0 then r f(x) j + y T A j = 0. Special Case: (Strong LP Duality) Let f(x) = -c T x. (So the convex program is max { c T x : Ax=b, x ¸ 0 }) Then x is optimal iff 9 y 2 R m s.t. 1) -c T + y T A ¸ 0, 2) For all j, if x j >0 then - c j + y T A j = 0. y is feasible for Dual LP Complementary Slackness holds

16. Theorem: Let f: R n ! R be a convex, C 2 function. Let x 2 R n be a feasible solution to the convex program min { f(x) : Ax=b, x ¸ 0 } Then x is optimal iff 9 y 2 R m s.t. 1) r f(x) T + y T A ¸ 0, 2) For all j, if x j >0 then r f(x) j + y T A j = 0. Proof: ( direction. Suppose such a y exists. Then ( r f(x) T + y T A) x = 0. (Just like complementary slackness) For any feasible z 2 R n, we have ( r f(x) T + y T A) z ¸ 0. Subtracting these, and using Ax=Az=b, we get r f(x) T (z-x) ¸ 0 8 feasible z. So x is optimal. (By earlier proposition Minimizing over a Convex Set)

17. Theorem: Let f: R n ! R be a convex, C 2 function. Let x 2 R n be a feasible solution to the convex program min { f(x) : Ax=b, x ¸ 0 } Then x is optimal iff 9 y 2 R m s.t. 1) r f(x) T + y T A ¸ 0, 2) For all j, if x j >0 then r f(x) j + y T A j = 0. Proof: ) direction. Suppose x is optimal. Let c=- r f(x). Then r f(x) T (z-x) ¸ 0 ) c T z · c T x for all feasible points z. By our earlier proposition Minimizing over a Convex Set

18. Theorem: Let f: R n ! R be a convex, C 2 function. Let x 2 R n be a feasible solution to the convex program min { f(x) : Ax=b, x ¸ 0 } Then x is optimal iff 9 y 2 R m s.t. 1) r f(x) T + y T A ¸ 0, 2) For all j, if x j >0 then r f(x) j + y T A j = 0. Proof: ) direction. Suppose x is optimal. Let c=- r f(x). Then r f(x) T (z-x) ¸ 0 ) c T z · c T x for all feasible points z. So x is optimal for the LP max { c T x : Ax=b, x ¸ 0 }. So there is an optimal solution y to dual LP min { b T y : A T y ¸ c }. So r f(x) T + y T A = -c T + y T A ¸ 0 ) (1) holds. Furthermore, x and y are both optimal so C.S. holds. ) whenever x j >0, the j th dual constraint is tight ) y T A j = c j ) (2) holds. ¥

19. Smallest Ball Problem Let P={p 1,…,p n } be points in R d. Let Q be d x n matrix s.t. Q i =p i. Let z 2 R n satisfy z i =p i T p i. Define f : R n ! R by f(x) = x T Q T Qx - x T z. Claim 1: f is convex. Consider the convex program min { f(x) : j x j = 1, x ¸ 0 }. Claim 2: This program has an optimal solution x. Claim 3: Let p * =Qx and r= -f(x). Then P ½ B(p *,r). Claim 4: B(p *,r) is the smallest ball containing P.

20. Let P={p 1,…,p n } be points in R d. Let Q be d x n matrix s.t. Q i =p i. Let z 2 R n satisfy z i =p i T p i. Define f : R n ! R by f(x) = x T Q T Qx - x T z. Claim 1: f is convex. Proof: By our earlier example, the Hessian of f at x is H(x) = 2 ¢ Q T Q. This is positive semi-definite. By our proposition Convexity and Hessian, f is convex. ¤

21. Let P={p 1,…,p n } be points in R d. Let Q be d x n matrix s.t. Q i =p i. Let z 2 R n satisfy z i =p i T p i. Define f : R n ! R by f(x) = x T Q T Qx - x T z. Consider the convex program min { f(x) : j x j = 1, x ¸ 0 }. Claim 2: This program has an optimal solution. Proof: The objective function is continuous. The feasible region is a bounded polyhedron, and hence compact. By Weierstrass Theorem, an optimal solution exists. ¤

22. Let Q be d x n matrix s.t. Q i =p i. Let z 2 R n satisfy z i =p i T p i. Define f : R n ! R by f(x) = x T Q T Qx - x T z. Let x be an optimal solution of min { f(x) : j x j =1, x ¸ 0 } Let p * = Qx and r 2 = -f(x) = j x j p j T p j - p *T p *. Claim 3: The ball B(p *,r) contains P. Proof: Note that r f(x) T = 2x T Q T Q - z T. By KKT, 9 y 2 R s.t. 2p j T p * - p j T p j + y ¸ 0 8 j. Furthermore, equality holds 8 j s.t. x j >0. So y = j x j y = j x j p j T p j - 2 j x j p j T p * = j x j p j T p j - 2p *T p *. So y + p *T p * = j x j p j T p j - p *T p * = -f(x) ) r = y + p *T p *

23. Let p * = Qx and r 2 = -f(x) = j x j p j T p j - p *T p *. Claim 3: The ball B(p *,r) contains P. Proof: Note that r f(x) = 2x T Q T Q - z. By KKT, 9 y 2 R s.t. 2p j T p * - p j T p j + y ¸ 0 8 j. Furthermore, equality holds 8 j s.t. x j >0. ) r = y + p *T p * It remains to show that B(p *,r) contains P. This holds iff k p j -p * k · r 8 j. Now k p j -p * k 2 = (p j -p * ) T (p j -p * ) = p *T p * -2p j T p * +p j T p j · y+p *T p * = r 2 8 j. ¤

24. Claim 4: B(p *,r) is the smallest ball containing P. Proof: See Textbook.