MIT and James Orlin © Nonlinear Programming Theory

MIT and James Orlin © Difficulties of NLP Models Nonlinear Programs: Linear Program:

MIT and James Orlin © Graphical Analysis of Non-linear programs in two dimensions: An example Minimize subject to (x - 8) 2 + (y - 9) 2  49 x  2 x  13 x + y  24

MIT and James Orlin © Where is the optimal solution? y x Note: the optimal solution is not at a corner point. It is where the isocontour first hits the feasible region.

MIT and James Orlin © Another example: y x Minimize (x-8) 2 + (y-8) 2 Then the global unconstrained minimum is also feasible. The optimal solution is not on the boundary of the feasible region.

MIT and James Orlin © Local vs. Global Optima There may be several locally optimal solutions. x z 1 0 z = f(x) max f(x) s.t. 0  x  1 A B C Def’n: Let x be a feasible solution, then –x is a global max if f(x)  f(y) for every feasible y. –x is a local max if f(x)  f(y) for every feasible y sufficiently close to x (i.e., x j -  y j  x j +  for all j and some small  ).

MIT and James Orlin © When is a locally optimal solution also globally optimal? For minimization problems –The objective function is convex. –The feasible region is convex.

W P Convexity and Extreme Points We say that a set S is convex, if for every two points x and y in S, and for every real number in [0,1], x + (1- )y  S. The feasible region of a linear program is convex. x y We say that an element w  S is an extreme point (vertex, corner point), if w is not the midpoint of any line segment contained in S. 8

MIT and James Orlin © On convex feasible regions If all constraints are linear, then the feasible region is convex

MIT and James Orlin © On Convex Feasible Regions The intersection of convex regions is convex

MIT and James Orlin © S Recognizing convex sets Rule of thumb: suppose for all x, y  S the midpoint of x and y is in S. Then S is convex. x y It is convex if the entire line segment is always in S. (x+y)/2

Which are convex? C B B  CB  CB  C D A 12

MIT and James Orlin © Joining two points on a curve The line segment joining two points on a curve. Let f( ) be a function, and let g( y+(1-  z)) = f(y) + (1-  )f(z) for 0  . f(y) f(z) (y+z)/2 f(y)/2 + f(z)/2 g(x) g(y) = f(y) g(z) = f(z) g(y/2 + z/2) = f(y)/2 + g(z)/2 y z f(x) x

MIT and James Orlin © Convex Functions Convex Functions: f( y + (1-  )z)  f(y) + (1-  )f(z) for every y and z and for 0 . e.g., = 1/2 f(y/2 + z/2)  f(y)/2 + f(z)/2 Line joining any points is above the curve f(x) x y z f(y) f(z) (y+z)/2 f(y)/2 + f(z)/2 We say “strict” convexity if sign is “<” for 0 .

MIT and James Orlin © Concave Functions Concave Functions: f( y + (1-  )z)  f(y) + (1-  )f(z) for every y and z and for 0 . e.g., = 1/2 f(y/2 + z/2)  f(y)/2 + f(z)/2 Line joining any points is below the curve x y z f(y) f(z) (y+z)/2 f(y)/2 + f(z)/2 We say “strict” concavity if sign is “<” for 0 .

MIT and James Orlin © Classify as convex or concave or both or neither.

MIT and James Orlin © More on convex functions x f(x) -x f(-x) If f(x) is convex, then f(-x) is convex.

MIT and James Orlin © More on convex functions x y If f(x) is convex, then K - f(x) is concave. x y

MIT and James Orlin © More on convex functions If f(x) is a twice differentiable function of one variable, and if f”(x) > 0 for all x, then f(x) is convex. f(x) = x 2. f’(x) = 2x, f”(x) = 2 f(x) = - ln(x) for x > 0 f’(x) = -1/x, f”(x) = 1/x 2

MIT and James Orlin © f(x)g(x) Even more on convex functions If f(x) is convex and g(x) is convex, then so is f(x) + g(x) f(x)+g(x)

MIT and James Orlin © f(x)g(x) Even more on convex functions If f(x) is convex and g(x) is convex, then so is max [f(x), g(x)] max[f(x), g(x)]

MIT and James Orlin © What functions are convex? f(x) = 4x + 7 all linear functions f(x) = 4x 2 – 13some quadratic functions f(x) = e x f(x) = 1/x for x > 0 f(x) = |x| f(x) = - ln(x) for x > 0

MIT and James Orlin © Convex functions vs. convex sets If y = f(x) is convex, then {(x,y) : f(x)  y} is a convex set y x f(x)

MIT and James Orlin © Local Minimum Property A local min of a convex function on a convex feasible region is also a global min. Strict convexity implies that the global minimum is unique. The following NLPs can be solved –Minimization Problems with a convex objective function and linear constraints

MIT and James Orlin © Local minimum property There is a unique local minimum for the function below. y x f(x) The local minimum is a global minimum

MIT and James Orlin © Local Maximum Property A local max of a concave function on a convex feasible region is also a global max. Strict concavity implies that the global optimum is unique. Given this, the following NLPs can be solved –Maximization Problems with a concave objective function and linear constraints

MIT and James Orlin © Local maximum property y x There is a unique local maximum for the function below. The local maximum is a global minimum

MIT and James Orlin © More on local optimality The techniques for non-linear optimization minimization usually find local optima. This is useful when a locally optimal solution is a globally optimal solution It is not so useful in many situations. Conclusion: if you solve an NLP, try to find out how good the local optimal solutions are.

MIT and James Orlin © Finding a local optimal for a single variable NLP Solving NLP's with One Variable: max f(  ) s.t. a  b Optimal solution is either a boundary point or satisfies f  (   ) = 0 and f  (   ) < 0.  f(  ) a  * b  f(  ) a  * b  f(  ) a  * b

MIT and James Orlin © Unimodal Functions A single variable function f is unimodal if there is at most one local maximum (or at most one local minimum).

MIT and James Orlin © Solving Single Variable NLP (contd.) If f(  ) is concave (or simply unimodal) and differentiableunimodal max f(  ) s.t. a    b Bisection (or Bolzano) Search: Step 1. Begin with the region of uncertainty for  as [a, b]. Evaluate f  (  ) at the midpoint   =(a+b)/2. Step 2. If f  (   ) > 0, then eliminate the interval up to  . If f  (   ) < 0, then eliminate the interval beyond  . Step 3. Evaluate f  (  ) at the midpoint of the new interval. Return to Step 2 until the interval of uncertainty is sufficiently small.

MIT and James Orlin © Interval Bisection or Bolzano Search 0 Determine by taking a derivative if a local maximum is to the right or left. to right to left

MIT and James Orlin ©

MIT and James Orlin © Other Search Techniques Instead of taking derivatives (which may be computationally intensive), use two function evaluations to determine updated interval. Fibonacci Search Step 1. Begin with the region of uncertainty for  as [a, b]. Evaluate f  (   ) and f  (   ) for 2 symmetric points   <  . Step 2. If f  (   )  f  (   ), then eliminate the interval up to  . If f  (   )  f  (   ), then eliminate the interval beyond  . Step 3. Select a second point symmetric to the point already in the new interval, rename these points   and   such that   <   and evaluate f  (   ) and f  (   ). Return to Step 2 until the interval is sufficiently small.

MIT and James Orlin © On Fibonacci search 1, 1, 2, 3, 5, 8, 13, 21, 34 At iteration 1, the length of the search interval is the kth fibonacci number for some k At iteration j, the length of the search interval is the k-j+1 fibonacci number. The technique converges to the optimal when the function is unimodal.

MIT and James Orlin © Finding a local maximum using Fibonacci Search Where the maximum may be Length of search interval

MIT and James Orlin © The search finds a local maximum, but not necessarily a global maximum

MIT and James Orlin © The search finds a local maximum, but not necessarily a global maximum

MIT and James Orlin © Number of function evaluations in Fibonacci Search As new point is chosen symmetrically, the length l k of successive search intervals is given by: l k = l k+1 + l k+2. Solving for these lengths given a final interval length of 1, l n = 1, gives the Fibonacci numbers: 1, 2, 3, 5, 8, 13, 21, 34,… Thus, if the initial interval has length 34, it takes 8 function calculations to reduce the interval length to 1. Remark: if the function is convex or unimodal, then fibonacci search converges to the global maximum

MIT and James Orlin © NLP Summary Convex and Concave functions as well as convex sets are important properties Bolzano and Fibonacci search techniques –used to solve single variable unimodal functions