1. Constrained Optimization Rong Jin

2. Outline  Equality constraints  Inequality constraints  Linear Programming  Quadratic Programming

3. Optimization Under Equality Constraints  Maximum Entropy Model  English ‘in’  French {dans (1), en (2), à (3), au cours de (4), pendant (5)}

4. Reducing variables  Representing variables using only p 1 and p 4  Objective function is changed  Solution: p 1 = 0.2, p 2 = 0.3, p 3 =0.1, p 4 = 0.2, p 5 = 0.2

5. Maximum Entropy Model for Classification  It is unlikely that we can use the previous simple approach to solve such a general  Solution: Lagrangian

6. Equality Constraints: Lagrangian  Introduce a Lagrange multiplier for the equality constraint  Construct the Lagrangian  Necessary condition A optimal solution for the original optimization problem has to be one of the stationary point of the Lagrangian

7. Example:  Introduce a Lagrange multiplier for constraint  Construct the Lagrangian  Stationary points

8. Lagrange Multipliers  Introducing a Lagrange multiplier for each constraint  Construct the Lagrangian for the original optimization problem

9. Lagrange Multiplier  We have more variables p 1, p 2, p 3, p 4, p 5 and, 1, 2, 3  Necessary condition (first order condition) A local/global optimum point for the original constrained optimization problem  a stationary point of the corresponding Lagrangian Original Entropy Function Constraints

10. Stationary Points for Lagrangian All probabilities p 1, p 2, p 3, p 4, p 5 are expressed as functions of Lagrange multipliers s

11. Dual Problem  p 1, p 2, p 3, p 4, p 5 are expressed as functions of s  We can even remove the variable 3  However, it is still difficult to obtain the solution s such that the constraints are satisfied  Dual problem Substitute the expression for ps into the Lagrangian Find the s that MINIMIZE the substituted Lagrangian

12. Dual Problem Finding s such that the above objective function is minimized Original Lagrangian Substituted Lagrangian Expression for ps

13. Dual Problem  Using dual problem Constrained optimization  unconstrained optimization  Need to change maximization to minimization  Only valid when the original optimization problem is convex/concave (strong duality) Dual Problem Primal Problem x*= * When convex/concave

14. Maximum Entropy Model for Classification  Introduce a Lagrange multiplier for each linear constraint

15. Maximum Entropy Model for Classification  Construct the Lagrangian for the original optimization problem Original Entropy Function Consistency Constraint Normalization Constraint

16. Stationary Points Stationary points: first derivatives are zero Sum of conditional probabilities must be one Conditional Exponential Model !

17. Support Vector Machine  Having many inequality constraints  Solving the above problem directly could be difficult  Many variables: w, b,   Unable to use nonlinear kernel function

18. Inequality Constraints: Modified Lagrangian  Introduce a Lagrange multiplier for the inequality constraint  Construct the Lagrangian  Kuhn-Tucker condition A optimal solution for the original optimization problem will satisfy the following conditions Non-negative Lagrange Multiplier Two cases: 1.g(x) = c, 2.g(x) > c  =0

19. Example:  Introduce a Lagrange multiplier for constraint  Construct the Lagrangian  KT conditions  Expressing objective function using  Solution is =3

20. SVM Model  Lagrange multipliers for inequality constraints Min  Max +  

21. SVM Model  Lagrangian for SVM model  Kuhn-Tucker condition

22. Dual Problem for SVM  Express w, b,  using  and   Finding solution satisfying KT conditions is difficult

23. Dual Problem for SVM  Rewrite the Lagrangian function using only  and   Simplify using KT conditions

24. Dual Problem for SVM  Final dual problem Maximize  Minimize

25. Quadratic Programming Find Subject to

26. Linear Programming Find Subject to Very very useful algorithm 1300+ papers 100+ books 10+ courses 100s of companies Main methods Simplex method Interior point method Most important: how to convert a general problem into the above standard form

27. Example  Need to change max to min Find Subject to

28. Example  Need change  to  Find Subject to

29. Example  Need to convert the inequality Find Subject to

30. Example  Need change |x 3 | Find Subject to