1. L11 Optimal Design L.Multipliers
Homework
Review
Convex sets, functions
Convex Programming Problem
Summary

2. Constrained Optimization LaGrange Multiplier Method
Remember:
Standard form
Max problems f(x) = - F(x)

3. KKT Necessary Conditions for Min
Regularity check - gradients of active inequality constraints are linearly independent

4. Prob 4.120

5. KKT Necessary Conditions

6. Case 1

7. Case 2 Regularity: 1. pt is feasible 2. only one active constraint
Point is KKT pt, OK!

8. Case 3

9. Case 4

10. Sensitivity or Case 2 Constraint Variation Sensitivity
From convexity theorems:
1. Constraints linear
2. Hf is PD
Therefore KKT Pt is global Min!

11. Graphical Solution 3 4 1 2

12. LaGrange Multiplier Method
May produce a KKT point
A KKT point is a CANDIDATE minimum
It may not be a local Min
If a point fails KKT conditions, we cannot guarantee anything….
The point may still be a minimum.
We need a SUFFICIENT condition

13. Recall Unconstrained MVO
For x* to be a local minimum:
1rst order
Necessary
Condition
2nd order
Sufficient
Condition
i.e. H(x*) must be positive definite

14. Considerations for Constrained Min?
Objective function
Differentiable, continuous i.e. smooth?
Hf(x) Positive definite
(i.e. convexity of f(x) )
Weierstrass theorem hints:
x closed and bounded
x contiguous or separated, pockets of points?
Constraints h(x) & g(x)
Define the constraint set, i.e. feasible region
Convex: sets, functions, constraint set and
Programming problem

15. Punchline (Theorem 4.10, pg 165)
The first-order KKT conditions are Necessary and Sufficient for a GLOBAL minimum….if:
1. f(x) is convex
Hf(x) Positive definite
2. x is defined as a convex feasible set.
Equality constraints must be linear
Inequality constraints must be convex
HINT: linear functions are convex!

16. Convex sets Convex set:
All pts in feasible region on a straight line(s).
Non-convex set
Pts on line are not in feasible region

17. Single variable
No “gaps” in feasible “region”

18. Multiple variables Fig 4.21
What if it were an equality constraint?
misprint

19. Figure 4.22 Convex function f(x)=x2
Bowl that holds water. .

20. Fig 4.23 Characterization of a convex function.

21. Test for Convex Function
Difficult to use above definition!
However, Thm 4.8 pg 163:
If the Hessian matrix of the function is PD ro PSD at all points in the set S, then it is convex.
PD… “strictly” convex, otherwise
PSD… “convex”

22. Theorem 4.9 S is convex if: 1. hi are linear
Given:
S is convex if:
1. hi are linear
2. gj are convex i.e. Hg PD or PSD
When f(x) and S are convex=
“convex programming problem”

23. “Sufficient” Theorem 4.10, pg 165
The first-order KKT conditions are Necessary and Sufficient for a GLOBAL minimum….if:
1. f(x) is convex
Hf(x) Positive definite
2. x is defined as a convex feasible set S
Equality constraints must be linear
Inequality constraints must be convex
HINT: linear functions are convex!

24. Summary
LaGrange multipliers are the instantaneous rate of change in f(x) w.r.t. relaxing a constraint.
KKT point is a CANDIDATE min!
(need sufficient conditions for proof)
Convex sets assure contiguity and or the smoothness of f(x)
KKT pt of a convex progamming problem is a GLOBAL MINIMUM!