CS B553: A LGORITHMS FOR O PTIMIZATION AND L EARNING Constrained optimization

K EY C ONCEPTS Constraint formulations Necessary optimality conditions Lagrange multipliers: equality constraints KKT conditions: equalities and inequalities

Objective function f Figure 1

S + feasible set S Objective function f Figure 1

S Local minima are either local minima of f or on the boundary of S Figure 2

S l1l1 u1u1 l2l2 u2u2 Bound constraints Linear inequalities A x  b S A i (row i ) bibi Linear equalities A x = b b1b1 A1A1 S General, nonlinear constraints Figure 3 h1(x)0h1(x)0 g 1 (x)=0 h2(x)0h2(x)0 S

Lagrange multipliers: one equality constraint At a local minimum (or maximum), the gradient of the objective and the constraint must be parallel Figure 4 g ( x )=0 x1x1 f(x1)f(x1) g(x1)g(x1) g(x2)g(x2) f(x2)f(x2) x2x2

If the constraint gradient and the objective gradient are not parallel, then there exists some direction v that you can move in to change f without changing g ( x ) Figure 5 f(x)f(x) x g(x)g(x) v

Interpretation : Suppose x* is a global minimum. I were to relax the constraint g ( x )=0 at a constant rate toward g ( x )=1, the value of tells me the rate of decrease of f ( x *). Figure 6 x*x*  f ( x * ) = -  g ( x * ) g(x*)g(x*)

One inequality constraint h ( x )  0. Either: Figure 7 h ( x ) < 0 h ( x ) > 0 h ( x ) = 0

One inequality constraint h ( x )  0. Either: 1. x is a critical point of f with h ( x ) < 0, or Figure 7 h ( x ) < 0 h ( x ) > 0 h ( x ) = 0  f ( x 1 )=0

Figure 7 h ( x ) < 0 h ( x ) > 0 h ( x ) = 0 x2x2 f(x2)f(x2) g(x2)g(x2) One inequality constraint h ( x )  0. Either: 1. x is a critical point of f with h ( x ) < 0, or 2. x is on boundary h ( x ) = 0 and satisfies a Lagrangian condition g(x3)g(x3) f(x3)f(x3) x3x3

Figure 8 h 1 ( x ) < 0 Multiple inequality constraints h 2 ( x ) < 0 x x