KKT Practice and Second Order Conditions from Nash and Sofer NLP KKT Practice and Second Order Conditions from Nash and Sofer

Unconstrained First Order Necessary Condition Second Order Necessary Second Order Sufficient

Easiest Problem Linear equality constraints

KKT Conditions Note for equality – multipliers are unconstrained Complementarity not an issue

Null Space Representation Let x* be a feasible point, Ax*=b. Any other feasible point can be written as x=x*+p where Ap=0 The feasible region {x : x*+p pN(A)} where N(A) is null space of A

Null and Range Spaces See Section 3.2 of Nash and Sofer for example

Orthogonality

Null Space Review

Constrained to Unconstrained You can convert any linear equality constrained optimization problem to an equivalent unconstrained problem Method 1 substitution Method 2 using Null space representation and a feasible point.

Example Solve by substitution becomes

Null Space Method x*= [4 0 0]’ x=x*+Zv becomes

General Method There exists a Null Space Matrix The feasible region is: Equivalent “Reduced” Problem

Optimality Conditions Assume feasible point and convert to null space formulation

Where is KKT? KKT implies null space Null Space implies KKT Gradient is not in Null(A), thus it must be in Range(A’)

Lemma 14.1 Necessary Conditions If x* is a local min of f over {x|Ax=b}, and Z is a null matrix Or equivalently use KKT Conditions

Lemma 14.2 Sufficient Conditions If x* satisfies (where Z is a basis matrix for Null(A)) then x* is a strict local minimizer

Lemma 14.2 Sufficient Conditions (KKT form) If (x*,*) satisfies (where Z is a basis matrix for Null(A)) then x* is a strict local minimizer

Lagrangian Multiplier * is called the Lagrangian Multiplier It represents the sensitivity of solution to small perturbations of constraints

Optimality conditions Consider min (x2+4y2)/2 s.t. x-y=10

Optimality conditions Find KKT point Check SOSC

In Class Practice Find a KKT point Verify SONC and SOSC

Linear Equality Constraints - I

Linear Equality Constraints - II

Linear Equality Constraints - III so SOSC satisfied, and x* is a strict local minimum Objective is convex, so KKT conditions are sufficient.

Next Easiest Problem Linear equality constraints Constraints form a polyhedron

Close to Equality Case Equality FONC: x* a2x = b a2x = b -a2 Polyhedron Ax>=b a3x = b a4x = b -a1 a1x = b x* contour set of function unconstrained minimum Which i are 0? What is the sign of I?

Close to Equality Case Equality FONC: x* a2x = b a2x = b -a2 Polyhedron Ax>=b a3x = b a4x = b -a1 a1x = b x* Which i are 0? What is the sign of I?

Inequality Case Inequality FONC: x* a2x = b a2x = b -a2 Polyhedron Ax>=b a3x = b a4x = b -a1 a1x = b x* Nonnegative Multipliers imply gradient points to the less than Side of the constraint.

Lagrangian Multipliers

Lemma 14.3 Necessary Conditions If x* is a local min of f over {x|Ax≤b}, and Z is a null-space matrix for active constraints then for some vector *

Lemma 14.5 Sufficient Conditions (KKT form) If (x*,*) satisfies

Lemma 14.5 Sufficient Conditions (KKT form) where Z+ is a basis matrix for Null(A +) and A + corresponds to nondegenerate active constraints) i.e.

Sufficient Example Find solution and verify SOSC

Linear Inequality Constraints - I

Linear Inequality Constraints - II

Linear Inequality Constraints - III

Linear Inequality Constraints - IV

Example Problem

You Try Solve the problem using above theorems:

Why Necessary and Sufficient? Sufficient conditions are good for? Way to confirm that a candidate point is a minimum (local) But…not every min satisifies any given SC Necessary tells you: If necessary conditions don’t hold then you know you don’t have a minimum. Under appropriate assumptions, every point that is a min satisfies the necessary cond. Good stopping criteria Algorithms look for points that satisfy Necessary conditions

General Constraints

Lagrangian Function Optimality conditions expressed using and Jacobian matrix were each row is a gradient of a constraint

Theorem 14.2 Sufficient Conditions Equality (KKT form) If (x*,*) satisfies

Theorem 14.4 Sufficient Conditions Inequality (KKT) If (x*,*) satisfies

Lemma 14.4 Sufficient Conditions (KKT form) where Z+ is a basis matrix for Null(A +) and A + corresponds to Jacobian of nondegenerate active constraints) i.e.

Sufficient Example Find solution and verify SOSC

Nonlinear Inequality Constraints - I

Nonlinear Inequality Constraints - II

Nonlinear Inequality Constraints - III

Sufficient Example Find solution and verify SOSC

Nonlinear Inequality Constraints - V

Nonlinear Inequality Constraints - VI

Theorem 14.1 Necessary Conditions- Equality If x* is a local min of f over {x|g(x)=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then

Theorem 14.3 Necessary Conditions If x* is a local min of f over {x|g(x)>=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then

Regular point If x* is a regular point with respect to the constraints g(x*) if the gradient of the active constraints are linearly independent. For equality constraints, all constraints are active so should have linearly independent rows.

Necessary Example Show optimal solution x*=[1,0]’ is regular and find KKT point

Constraint Qualifications Regularity is an example of a constraint qualification CQ. The KKT conditions are based on linearizations of the constraints. CQ guarantees that this linearization is not getting us into trouble. Problem is KKT point might not exist. There are many other CQ,e.g., for inequalities Slater is there exists g(x)<0. Note CQ not needed for linear constraints.

KKT Summary X* is global min Convex f Convex constraints X* is local min CQ SOSC KKT Satisfied