Constrained Optimization Rong Jin

Outline  Equality constraints  Inequality constraints  Linear Programming  Quadratic Programming

Optimization Under Equality Constraints  Maximum Entropy Model  English ‘in’  French {dans (1), en (2), à (3), au cours de (4), pendant (5)}

Reducing variables  Representing variables using only p 1 and p 4  Objective function is changed  Solution: p 1 = 0.2, p 2 = 0.3, p 3 =0.1, p 4 = 0.2, p 5 = 0.2

Maximum Entropy Model for Classification  It is unlikely that we can use the previous simple approach to solve such a general  Solution: Lagrangian

Equality Constraints: Lagrangian  Introduce a Lagrange multiplier for the equality constraint  Construct the Lagrangian  Necessary condition A optimal solution for the original optimization problem has to be one of the stationary point of the Lagrangian

Example:  Introduce a Lagrange multiplier for constraint  Construct the Lagrangian  Stationary points

Lagrange Multipliers  Introducing a Lagrange multiplier for each constraint  Construct the Lagrangian for the original optimization problem

Lagrange Multiplier  We have more variables p 1, p 2, p 3, p 4, p 5 and, 1, 2, 3  Necessary condition (first order condition) A local/global optimum point for the original constrained optimization problem  a stationary point of the corresponding Lagrangian Original Entropy Function Constraints

Stationary Points for Lagrangian All probabilities p 1, p 2, p 3, p 4, p 5 are expressed as functions of Lagrange multipliers s

Dual Problem  p 1, p 2, p 3, p 4, p 5 are expressed as functions of s  We can even remove the variable 3  However, it is still difficult to obtain the solution s such that the constraints are satisfied  Dual problem Substitute the expression for ps into the Lagrangian Find the s that MINIMIZE the substituted Lagrangian

Dual Problem Finding s such that the above objective function is minimized Original Lagrangian Substituted Lagrangian Expression for ps

Dual Problem  Using dual problem Constrained optimization  unconstrained optimization  Need to change maximization to minimization  Only valid when the original optimization problem is convex/concave (strong duality) Dual Problem Primal Problem x*= * When convex/concave

Maximum Entropy Model for Classification  Introduce a Lagrange multiplier for each linear constraint

Maximum Entropy Model for Classification  Construct the Lagrangian for the original optimization problem Original Entropy Function Consistency Constraint Normalization Constraint

Stationary Points Stationary points: first derivatives are zero Sum of conditional probabilities must be one Conditional Exponential Model !

Support Vector Machine  Having many inequality constraints  Solving the above problem directly could be difficult  Many variables: w, b,   Unable to use nonlinear kernel function

Inequality Constraints: Modified Lagrangian  Introduce a Lagrange multiplier for the inequality constraint  Construct the Lagrangian  Kuhn-Tucker condition A optimal solution for the original optimization problem will satisfy the following conditions Non-negative Lagrange Multiplier Two cases: 1.g(x) = c, 2.g(x) > c  =0

Example:  Introduce a Lagrange multiplier for constraint  Construct the Lagrangian  KT conditions  Expressing objective function using  Solution is =3

SVM Model  Lagrange multipliers for inequality constraints Min  Max +  

SVM Model  Lagrangian for SVM model  Kuhn-Tucker condition

Dual Problem for SVM  Express w, b,  using  and   Finding solution satisfying KT conditions is difficult

Dual Problem for SVM  Rewrite the Lagrangian function using only  and   Simplify using KT conditions

Dual Problem for SVM  Final dual problem Maximize  Minimize

Quadratic Programming Find Subject to

Linear Programming Find Subject to Very very useful algorithm papers 100+ books 10+ courses 100s of companies Main methods Simplex method Interior point method Most important: how to convert a general problem into the above standard form

Example  Need to change max to min Find Subject to

Example  Need change  to  Find Subject to

Example  Need to convert the inequality Find Subject to

Example  Need change |x 3 | Find Subject to