1. 1 15.053 Thursday, April 25 Nonlinear Programming Theory Separable programming Handouts: Lecture Notes

2. 2 Difficulties of NLP Models Linear Program: Nonlinear Programs:

3. 3 Graphical Analysis of Non-linear programs in two dimensions: An example Minimize subject to(x - 8) 2 + (y - 9) 2 ≤ 49 x  2 x  13 x + y  24

4. 4 Where is the optimal solution? Note: the optimal solution is not at a corner point. It is where the isocontour first hits the feasible region.

5. 5 Another example: Minimize (x-8) 2 + (y-8) 2 Then the global unconstrained minimum is also feasible. The optimal solution is not on the boundary of the feasible region.

6. 6 Local vs. Global Optima Def’n: Let x be a feasible solution, then – x is a global max if f(x) ≥ f(y) for every feasible y. – x is a local max if f(x) ≥ f(y) for every feasible y sufficiently close to x (i.e., xj-ε ≤ yj ≤ xj+ ε for all j and some small ε). There may be several locally optimal solutions.

7. 7 When is a locally optimal solution also globally optimal? We are minimizing. The objective function is convex. The feasible region is convex.

8. 8 Convexity and Extreme Points We say that a set S is convex, if for every two points x and y in S, and for every real number λ in [0,1], λx + (1-λ)y ε S. The feasible region of a linear program is convex. We say that an element w ε S is an extreme point (vertex, corner point), if w is not the midpoint of any line segment contained in S.

9. 9 Recognizing convex feasible regions If all constraints are linear, then the feasible region is convex The intersection of convex regions is convex If for all feasible x and y, the midpoint of x and y is feasible, then the region is convex (except in totally non-realistic examples. )

10. 10 Which are convex?

11. 11 Convex Functions Convex Functions: f( y + (1- )z)  f(y) + (1- )f(z) for every y and z and for 0 . e.g., f((y+z)/2)  f(y)/2 + f(z)/2 We say “strict” convexity if sign is “<” for 0 . Line joining any points is above the curve

12. 12 Convex Functions Convex Functions: f( y + (1- )z) ≥ f(y) + (1- )f(z) for every y and z and for 0 . e.g., f((y+z)/2) ≥ f(y)/2 + f(z)/2 We say “strict” convexity if sign is “<” for 0 . Line joining any points is above the curve

13. 13 Classify as convex or concave or both or neither.

14. 14 What functions are convex? f(x) = 4x + 7 all linear functions f(x) = 4x 2 – 13 some quadratic functions f(x) = e x f(x) = 1/x for x > 0 f(x) = |x| f(x) = - ln(x) for x > 0 Sufficient condition: f”(x) > 0 for all x.

15. 15 What functions are convex? If f(x) is convex, and g(x) is convex. Then so is h(x) = a f(x) + b g(x) for a>0, b>0. If y = f(x) is convex, then {(x,y) : f(x) ≤ y} is a convex set

16. 16 Local Maximum (Minimum) Property A local max of a concave function on a convex feasible region is also a global max. A local min of a convex function on a convex feasible region is also a global min. Strict convexity or concavity implies that the global optimum is unique. Given this, we can exactly solve: – Maximization Problems with a concave objective function and linear constraints – Minimization Problems with a convex objective function and linear constraints

17. 17 Which are convex feasible regions? (x, y) : y ≤ x 2 + 2 (x, y) : y ≥ x 2 + 2 (x,y) : y = x 2 + 2

18. 18 More on local optimality The techniques for non-linear optimization minimization usually find local optima. This is useful when a locally optimal solution is a globally optimal solution It is not so useful in many situations. Conclusion: if you solve an NLP, try to find out how good the local optimal solutions are.

19. 19 Solving NLP’s by Excel Solver

20. 20 Finding a local optimal for a single variable NLP Solving NLP's with One Variable: max f(θ) s.t. a ≤ θ ≤ b Optimal solution is either a boundary point or satisfies f′(θ ∗ ) = 0 and f ″(θ ∗ ) < 0.

21. 21 Solving Single Variable NLP (contd.) If f(θ) is concave (or simply unimodal) and differentiable max f(θ) s.t. a ≤ θ ≤ b Bisection (or Bolzano) Search: Step 1. Begin with the region of uncertainty for θ as [a, b]. Evaluate f ′ (θ) at the midpoint θΜ =(a+b)/2. Step 2. If f ′ (θΜ) > 0, then eliminate the interval up to θΜ. If f′(θΜ) < 0, then eliminate the interval beyond θΜ. Step 3. Evaluate f′(θ) at the midpoint of the new interval. Return to Step 2 until the interval of uncertainty is sufficiently small.

22. 22 Unimodal Functions A single variable function f is unimodal if there is at most one local maximum (or at most one local minimum).

23. 23 Other Search Techniques Instead of taking derivatives (which may be computationally intensive), use two function evaluations to determine updated interval. Fibonacci Search Step 1. Begin with the region of uncertainty for θ as [a, b]. Evaluate f (θ1) and f (θ2) for 2 symmetric points θ1<θ2. Step 2. If f (θ1) ≤ f (θ2), then eliminate the interval up tθ1. If f (θ1) > f (θ2), then eliminate the interval beyond θ2. Step 3. Select a second point symmetric to the point already in the new interval, rename these points θ1 and θ2 such that θ1<θ2 and evaluate f (θ1) and f (θ2). Return to Step 2 until the interval is sufficiently small.

24. 24 On Fibonacci search 1, 1, 2, 3, 5, 8, 13, 21, 34 At iteration 1, the length of the search interval is the kth fibonacci number for some k At iteration j, the length of the search interval is the k-j+1 fibonacci number. The technique converges to the optimal when the function is unimodal.

25. 25 Finding a local maximum using Fibonacci Search. Length of search interval Where the maximum may be

26. 26 The search finds a local maximum, but not necessarily a global maximum.

27. 27 The search finds a local maximum, but not necessarily a global maximum.

28. 28 Number of function evaluations in Fibonacci Search As new point is chosen symmetrically, the length lk of successive search intervals is given by: lk= lk+1 + lk+2. Solving for these lengths given a final interval length of 1, ln = 1, gives the Fibonacci numbers: 1, 2, 3, 5, 8, 13, 21, 34,… Thus, if the initial interval has length 34, it takes 8 function calculations to reduce the interval length to 1. Remark: if the function is convex or unimodal, then fibonacci search converges to the global maximum

29. 29 Separable Programming Separable programs have the form: Max st Each variable xj appears separately, one in each function gij and one in each function fj in the objective. Each non-linear function is of a single variable.

30. 30 Examples of Separable Programs f(x 1,x 2 ) = x 1 (30-x 1 )+x 2 (35-x 2 )-2x 1 2 -3x 2 2 f(x 1,x 2 ) = x 1 5 + -18e -x2 -4x 2 3 x1x1 f(x 1,x 2,x 2 ) = lnx 1 5 -sinx 2 -x 3 e -x3 )+7x 1 -4

31. 31 Approximating a non-linear function with a piecewise linear function. Aspect 1. Choosing the approximation. Aspect 2. When is the piecewise linear approximation a linear program in disguise?

32. 32 Approximating a non-linear function of 1 variable

33. 33 Approximating a non-linear function of 1 variable: the λ method Choose different values of x to approximate the x-axis Approximate using piecewise linear segments

34. 34 More on the  method Suppose that for –3 ≤ x ≤ -1, we represent x hasλ 1 (-3) +λ 2 (-1) whereλ 1 +λ 2 = 1 andλ 1,λ 2 ≥ 0 Then we approximate f(x) as λ 1 (-20) +λ 2 (-7 1/3) a 1 = -3, f(a 1 ) = -20 a 2 = -1 f(a 2 ) = -7 1/3

35. 35 More on the  method a 2 = -1, f(a 2 ) = -7 1 /3 a 3 = 1 f(a 3 ) = -2 2/3 Suppose that for –1 ≤ x ≤ 1, we represent x hasλ 2 (-3) +λ 3 (-1) whereλ 2 +λ 3 = 1 andλ 2,λ 3 ≥ 0 How do we approximate f( ) in this interval? What if –3 ≤ x ≤ 1?

36. 36 Almost the  method Original problem: min x 3 /3 + 2x – 5 + more terms s.t. -3 ≤ x ≤ 3 + many more constraints a1 = -3; a2 = - 1; a3 = 1; a4 = 3 f(a1) = -20; f(a2) = -7 1/3; f(a3) = -2 2/3; f(a4) = 4 Approximated problem: minλ 1 f(a 1 ) +λ 2 f(a 2 ) +λ 3 f(a 3 ) +λ 4 f(a 4 ) more linear terms s.t. λ 1 + λ 2 + λ 3 + λ 4 = 1 ; λ ≥ 0 + many more constraints

37. 37 Why the approximation is incorrect Approximated problem: minλ 1 f(a 1 ) +λ 2 f(a 2 ) +λ 3 f(a 3 ) +λ 4 f(a 4 ) more linear terms s.t. λ 1 +λ 2 +λ 3 +λ 4 = 1 ; λ ≥ 0 Considerλ 1 = 1/2 ;λ 2 =0 ;λ 3 =1/2 ;λ 4 = 0; The method gives the correct approximation if only two consecutiveλs are positive.

38. 38 Adjacency Condition 1. At most two weights (λs) are positive 2. If exactly two weights (λs) are positive, then they are λj and λj+1 for some j 3. The same condition applies to every approximated function.

39. 39 Approximating a non-linear objective function for a minimization NLP. original problem: minimize {f(y): y ∈  P} Suppose that y =  j  j a j, where  j  j = 1 and  >= 0. Approximate f(y). minimize {  j  j f(a j ):  j  j a j ∈  P} Note: when given a choice of representing y in alternative ways, the LP will choose one that leads to the least objective value for the approximation.

40. 40 For minimizing a convex function, the  -method automatically satisfies the additional adjacency property. min Z =λ 1 f(a 1 ) +λ 2 f(a 2 ) +λ 3 f(a 3 ) +λ 4 f(a 4 )+λ 5 f(a 5 ) s.t. λ 1 + λ 2 + λ 3 + λ 4 +λ 5 = 1;  0 + adjacency condition + other constraints

41. 41 Feasible approximated objective functions without the adjacency conditions min Z =λ 1 f(a 1 ) +λ 2 f(a 2 ) +λ 3 f(a 3 ) +λ 4 f(a 4 )+λ 5 f(a 5 ) s.t. λ 1 + λ 2 + λ 3 + λ 4 +λ 5 = 1;  0 + other constraints

42. 42 But a minimum in this case always occurs on the piecewise linear curve. min Z =λ 1 f(a 1 ) +λ 2 f(a 2 ) +λ 3 f(a 3 ) +λ 4 f(a 4 )+λ 5 f(a 5 ) s.t. λ 1 + λ 2 + λ 3 + λ 4 +λ 5 = 1;  0 + other constraints

43. 43 Separable Programming (in the case of linear constraints) Begin with an NLP: Transform to Separable: Approximate using the  Method: Max f ( )x s.t. Dx  d x  0 Max ∑  f j (x j )  s.t. Dx  d x  0 n j=1

44. 44 Approximation Re-express in terms of  variables: If the original problem is concave, then you can drop the adjacency conditions (they are automatically satisfied.) and the adjacency conditions

45. 45 How can one construct separable functions? Term Substitution Constraints Restriction None

46. 46 Transformation Examples Ex : (x 1 + x 2 + x 3 ) 6 Substitute y 6 and let y = x 1 + x 2 + x 3 x1x22x1x22 1+ x 3 Ex : let y 1 and let y 2 and add the constraint

47. 47 NLP Summary Convex and Concave functions as well as convex sets are important properties Bolzano and Fibonacci search techniques – used to solve single variable unimodal functions Separable Programming – nonlinear objective and nonlinear constraints that are separable – General approximation technique