L11 Optimal Design L.Multipliers Homework Review Convex sets, functions Convex Programming Problem Summary

Constrained Optimization LaGrange Multiplier Method Remember: Standard form Max problems f(x) = - F(x)

KKT Necessary Conditions for Min Regularity check - gradients of active inequality constraints are linearly independent

Prob 4.120

KKT Necessary Conditions

Case 1

Case 2 Regularity: 1. pt is feasible 2. only one active constraint Point is KKT pt, OK!

Case 3

Case 4

Sensitivity or Case 2 Constraint Variation Sensitivity From convexity theorems: 1. Constraints linear 2. Hf is PD Therefore KKT Pt is global Min!

Graphical Solution 3 4 1 2

LaGrange Multiplier Method May produce a KKT point A KKT point is a CANDIDATE minimum It may not be a local Min If a point fails KKT conditions, we cannot guarantee anything…. The point may still be a minimum. We need a SUFFICIENT condition

Recall Unconstrained MVO For x* to be a local minimum: 1rst order Necessary Condition 2nd order Sufficient Condition i.e. H(x*) must be positive definite

Considerations for Constrained Min? Objective function Differentiable, continuous i.e. smooth? Hf(x) Positive definite (i.e. convexity of f(x) ) Weierstrass theorem hints: x closed and bounded x contiguous or separated, pockets of points? Constraints h(x) & g(x) Define the constraint set, i.e. feasible region Convex: sets, functions, constraint set and Programming problem

Punchline (Theorem 4.10, pg 165) The first-order KKT conditions are Necessary and Sufficient for a GLOBAL minimum….if: 1. f(x) is convex Hf(x) Positive definite 2. x is defined as a convex feasible set. Equality constraints must be linear Inequality constraints must be convex HINT: linear functions are convex!

Convex sets Convex set: All pts in feasible region on a straight line(s). Non-convex set Pts on line are not in feasible region

Single variable No “gaps” in feasible “region”

Multiple variables Fig 4.21 What if it were an equality constraint? misprint

Figure 4.22 Convex function f(x)=x2 Bowl that holds water. .

Fig 4.23 Characterization of a convex function.

Test for Convex Function Difficult to use above definition! However, Thm 4.8 pg 163: If the Hessian matrix of the function is PD ro PSD at all points in the set S, then it is convex. PD… “strictly” convex, otherwise PSD… “convex”

Theorem 4.9 S is convex if: 1. hi are linear Given: S is convex if: 1. hi are linear 2. gj are convex i.e. Hg PD or PSD When f(x) and S are convex= “convex programming problem”

“Sufficient” Theorem 4.10, pg 165 The first-order KKT conditions are Necessary and Sufficient for a GLOBAL minimum….if: 1. f(x) is convex Hf(x) Positive definite 2. x is defined as a convex feasible set S Equality constraints must be linear Inequality constraints must be convex HINT: linear functions are convex!

Summary LaGrange multipliers are the instantaneous rate of change in f(x) w.r.t. relaxing a constraint. KKT point is a CANDIDATE min! (need sufficient conditions for proof) Convex sets assure contiguity and or the smoothness of f(x) KKT pt of a convex progamming problem is a GLOBAL MINIMUM!