1. MIT and James Orlin © 2003 1 Nonlinear Programming Theory

2. MIT and James Orlin © 2003 2 Difficulties of NLP Models Nonlinear Programs: Linear Program:

3. MIT and James Orlin © 2003 3 Graphical Analysis of Non-linear programs in two dimensions: An example Minimize subject to (x - 8) 2 + (y - 9) 2  49 x  2 x  13 x + y  24

4. MIT and James Orlin © 2003 4 Where is the optimal solution? 024681012141618 0 2 4 6 8 10 12 14 16 18 y x Note: the optimal solution is not at a corner point. It is where the isocontour first hits the feasible region.

5. MIT and James Orlin © 2003 5 Another example: 024681012141618 0 2 4 6 8 10 12 14 16 18 y x Minimize (x-8) 2 + (y-8) 2 Then the global unconstrained minimum is also feasible. The optimal solution is not on the boundary of the feasible region.

6. MIT and James Orlin © 2003 6 Local vs. Global Optima There may be several locally optimal solutions. x z 1 0 z = f(x) max f(x) s.t. 0  x  1 A B C Def’n: Let x be a feasible solution, then –x is a global max if f(x)  f(y) for every feasible y. –x is a local max if f(x)  f(y) for every feasible y sufficiently close to x (i.e., x j -  y j  x j +  for all j and some small  ).

7. MIT and James Orlin © 2003 7 When is a locally optimal solution also globally optimal? For minimization problems –The objective function is convex. –The feasible region is convex.

8. W P 2468101214 2468101214 Convexity and Extreme Points We say that a set S is convex, if for every two points x and y in S, and for every real number in [0,1], x + (1- )y  S. The feasible region of a linear program is convex. x y We say that an element w  S is an extreme point (vertex, corner point), if w is not the midpoint of any line segment contained in S. 8

9. MIT and James Orlin © 2003 9 On convex feasible regions If all constraints are linear, then the feasible region is convex

10. MIT and James Orlin © 2003 10 On Convex Feasible Regions The intersection of convex regions is convex

11. MIT and James Orlin © 2003 11 S Recognizing convex sets Rule of thumb: suppose for all x, y  S the midpoint of x and y is in S. Then S is convex. x y It is convex if the entire line segment is always in S. (x+y)/2

12. Which are convex? C B B  CB  CB  C D A 12

13. MIT and James Orlin © 2003 13 Joining two points on a curve The line segment joining two points on a curve. Let f( ) be a function, and let g( y+(1-  z)) = f(y) + (1-  )f(z) for 0  . f(y) f(z) (y+z)/2 f(y)/2 + f(z)/2 g(x) g(y) = f(y) g(z) = f(z) g(y/2 + z/2) = f(y)/2 + g(z)/2 y z f(x) x

14. MIT and James Orlin © 2003 14 Convex Functions Convex Functions: f( y + (1-  )z)  f(y) + (1-  )f(z) for every y and z and for 0 . e.g., = 1/2 f(y/2 + z/2)  f(y)/2 + f(z)/2 Line joining any points is above the curve f(x) x y z f(y) f(z) (y+z)/2 f(y)/2 + f(z)/2 We say “strict” convexity if sign is “<” for 0 .

15. MIT and James Orlin © 2003 15 Concave Functions Concave Functions: f( y + (1-  )z)  f(y) + (1-  )f(z) for every y and z and for 0 . e.g., = 1/2 f(y/2 + z/2)  f(y)/2 + f(z)/2 Line joining any points is below the curve x y z f(y) f(z) (y+z)/2 f(y)/2 + f(z)/2 We say “strict” concavity if sign is “<” for 0 .

16. MIT and James Orlin © 2003 16 Classify as convex or concave or both or neither.

17. MIT and James Orlin © 2003 17 More on convex functions x f(x) -x f(-x) If f(x) is convex, then f(-x) is convex.

18. MIT and James Orlin © 2003 18 More on convex functions x y If f(x) is convex, then K - f(x) is concave. x y

19. MIT and James Orlin © 2003 19 More on convex functions If f(x) is a twice differentiable function of one variable, and if f”(x) > 0 for all x, then f(x) is convex. f(x) = x 2. f’(x) = 2x, f”(x) = 2 f(x) = - ln(x) for x > 0 f’(x) = -1/x, f”(x) = 1/x 2

20. MIT and James Orlin © 2003 20 f(x)g(x) Even more on convex functions If f(x) is convex and g(x) is convex, then so is f(x) + g(x) f(x)+g(x)

21. MIT and James Orlin © 2003 21 f(x)g(x) Even more on convex functions If f(x) is convex and g(x) is convex, then so is max [f(x), g(x)] max[f(x), g(x)]

22. MIT and James Orlin © 2003 22 What functions are convex? f(x) = 4x + 7 all linear functions f(x) = 4x 2 – 13some quadratic functions f(x) = e x f(x) = 1/x for x > 0 f(x) = |x| f(x) = - ln(x) for x > 0

23. MIT and James Orlin © 2003 23 Convex functions vs. convex sets If y = f(x) is convex, then {(x,y) : f(x)  y} is a convex set y x f(x)

24. MIT and James Orlin © 2003 24 Local Minimum Property A local min of a convex function on a convex feasible region is also a global min. Strict convexity implies that the global minimum is unique. The following NLPs can be solved –Minimization Problems with a convex objective function and linear constraints

25. MIT and James Orlin © 2003 25 Local minimum property There is a unique local minimum for the function below. y x f(x) The local minimum is a global minimum

26. MIT and James Orlin © 2003 26 Local Maximum Property A local max of a concave function on a convex feasible region is also a global max. Strict concavity implies that the global optimum is unique. Given this, the following NLPs can be solved –Maximization Problems with a concave objective function and linear constraints

27. MIT and James Orlin © 2003 27 Local maximum property y x There is a unique local maximum for the function below. The local maximum is a global minimum

28. MIT and James Orlin © 2003 28 More on local optimality The techniques for non-linear optimization minimization usually find local optima. This is useful when a locally optimal solution is a globally optimal solution It is not so useful in many situations. Conclusion: if you solve an NLP, try to find out how good the local optimal solutions are.

29. MIT and James Orlin © 2003 29 Finding a local optimal for a single variable NLP Solving NLP's with One Variable: max f(  ) s.t. a  b Optimal solution is either a boundary point or satisfies f  (   ) = 0 and f  (   ) < 0.  f(  ) a  * b  f(  ) a  * b  f(  ) a  * b

30. MIT and James Orlin © 2003 30 Unimodal Functions A single variable function f is unimodal if there is at most one local maximum (or at most one local minimum).

31. MIT and James Orlin © 2003 31 Solving Single Variable NLP (contd.) If f(  ) is concave (or simply unimodal) and differentiableunimodal max f(  ) s.t. a    b Bisection (or Bolzano) Search: Step 1. Begin with the region of uncertainty for  as [a, b]. Evaluate f  (  ) at the midpoint   =(a+b)/2. Step 2. If f  (   ) > 0, then eliminate the interval up to  . If f  (   ) < 0, then eliminate the interval beyond  . Step 3. Evaluate f  (  ) at the midpoint of the new interval. Return to Step 2 until the interval of uncertainty is sufficiently small.

32. MIT and James Orlin © 2003 32 6432162824 Interval Bisection or Bolzano Search 0 Determine by taking a derivative if a local maximum is to the right or left. to right to left

33. MIT and James Orlin © 2003 33

34. MIT and James Orlin © 2003 34 Other Search Techniques Instead of taking derivatives (which may be computationally intensive), use two function evaluations to determine updated interval. Fibonacci Search Step 1. Begin with the region of uncertainty for  as [a, b]. Evaluate f  (   ) and f  (   ) for 2 symmetric points   <  . Step 2. If f  (   )  f  (   ), then eliminate the interval up to  . If f  (   )  f  (   ), then eliminate the interval beyond  . Step 3. Select a second point symmetric to the point already in the new interval, rename these points   and   such that   <   and evaluate f  (   ) and f  (   ). Return to Step 2 until the interval is sufficiently small.

35. MIT and James Orlin © 2003 35 On Fibonacci search 1, 1, 2, 3, 5, 8, 13, 21, 34 At iteration 1, the length of the search interval is the kth fibonacci number for some k At iteration j, the length of the search interval is the k-j+1 fibonacci number. The technique converges to the optimal when the function is unimodal.

36. MIT and James Orlin © 2003 36 Finding a local maximum using Fibonacci Search. 21 13 26 18 1619 0 34 Where the maximum may be Length of search interval 342113853

37. MIT and James Orlin © 2003 37 The search finds a local maximum, but not necessarily a global maximum. 21 1326 18 1619 0 34

38. MIT and James Orlin © 2003 38 The search finds a local maximum, but not necessarily a global maximum. 21 1326 18 1619 0 34

39. MIT and James Orlin © 2003 39 Number of function evaluations in Fibonacci Search As new point is chosen symmetrically, the length l k of successive search intervals is given by: l k = l k+1 + l k+2. Solving for these lengths given a final interval length of 1, l n = 1, gives the Fibonacci numbers: 1, 2, 3, 5, 8, 13, 21, 34,… Thus, if the initial interval has length 34, it takes 8 function calculations to reduce the interval length to 1. Remark: if the function is convex or unimodal, then fibonacci search converges to the global maximum

40. MIT and James Orlin © 2003 40 NLP Summary Convex and Concave functions as well as convex sets are important properties Bolzano and Fibonacci search techniques –used to solve single variable unimodal functions