Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 1 Part 2 - Chapter 7 Optimization

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 2 Optimization Root finding and optimization are related, both involve –guessing and searching for a point on a function.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Fundamental difference is: Root finding is searching for zeros of a function or functions Optimization is finding the minimum or the maximum of a function of one or more variables. 3

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 4 figure 7.2

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 5 We’ll be looking at strategies to find the minimum of our functions of interest

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 6 Functions of more than one variable are harder to visualize

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 7

8 Each contour represents a constant value of f(x,y) We will be looking at strategies for both one- dimensional optimization and multi- dimensional optimization

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 9 Mathematical Background An optimization or mathematical programming problem generally can be stated as: Find x, which minimizes or maximizes f(x) subject to Where x is an n-dimensional design vector, f(x) is the objective function, d i (x) are inequality constraints, e i (x) are equality constraints, and a i and b i are constants

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 10 Optimization problems can be classified on the basis of the form of f(x): –If f(x) and the constraints are linear, we have linear programming. –If f(x) is quadratic and the constraints are linear, we have quadratic programming. –If f(x) is not linear or quadratic and/or the constraints are nonlinear, we have nonlinear programming. When equations are included, we have a constrained optimization problem; otherwise, it is unconstrained optimization problem.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 11 One-Dimensional Unconstrained Optimization If you don’t put a limit on x it’s hard to tell where the min’s or max’s occur In multimodal functions, both local and global optima can occur. Usually, we are interested in finding the absolute highest or lowest value of a function.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 12 How do we distinguish global optimum from local ones? By graphing to gain insight into the behavior of the function. Using randomly generated starting guesses and picking the largest of the optima as global. Perturbing the starting point to see if the routine returns a better point or the same local minimum.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example 7.1 If an object such as a bungee jumper is projected upward at a specified velocity – and if it is subjected to linear drag – its altitude as a function of time can be computed as: 13

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Determine the time and magnitude of the peak elevation 14 Given g = 9.81 m/s 2 z o =100 m v o = 55 m/s m = 80 kg c = 15 kg/s

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 15 Equation for the position of a bungie jumper – Eq. 7.1

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 16

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 17

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 18 Golden-Section Search A unimodal function has a single maximum or a minimum in the a given interval. For a unimodal function: –First pick two points that will bracket your extremum [x L, x u ]. –Pick an additional third point within this interval to determine whether a minimum (or maximum) occurred. –Then pick a fourth point to determine whether the minimum (or maximum) has occurred within the first three or last three points –The key is making this approach efficient by choosing intermediate points wisely thus minimizing the function evaluations by replacing the old values with new values.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. –First pick two points that will bracket your extremum [x L, x u ]. –Pick an additional third point within this interval to determine whether a minimum (or maximum) occurred. –Then pick a fourth point to determine whether the minimum (or maximum) has occurred within the first three or last three points 19

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. –First pick two points that will bracket your extremum [x L, x u ]. –Pick an additional third point within this interval to determine whether a minimum (or maximum) occurred. –Then pick a fourth point to determine whether the minimum (or maximum) has occurred within the first three or last three points 20

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. –First pick two points that will bracket your extremum [x L, x u ]. –Pick an additional third point within this interval to determine whether a minimum (or maximum) occurred. –Then pick a fourth point to determine whether the minimum (or maximum) has occurred within the first three or last three points 21 Eliminate Minimum

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. –First pick two points that will bracket your extremum [x L, x u ]. –Pick an additional third point within this interval to determine whether a minimum (or maximum) occurred. –Then pick a fourth point to determine whether the minimum (or maximum) has occurred within the first three or last three points 22 Eliminate Minimum Old x 2 Old x 1 New x 2 x1x1 xuxu xLxL

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Golden Ratio 23

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 24 The first condition specifies that the sum of the two sub lengths l 1 and l 2 must equal the original interval length. The second says that the ratio of the lengths must be equal l0 l0

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 25 Golden Ratio

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 26 Golden Ratio

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 27 d=R*(x u –x L )

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 28 The method starts with two initial guesses, x L and x u, that bracket one local extremum of f(x): Next two interior points x 1 and x 2 are chosen according to the golden ratio The function is evaluated at these two interior points.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 29 Two results can occur: If f(x 1 )<f(x 2 ) then the domain of x to the left of x 2 from x L to x 2, can be eliminated because it does not contain the minimum. Then, x 2 becomes the new x L for the next round.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 30 Two results can occur: If f(x 1 )<f(x 2 ) then the domain of x to the left of x 2 from x L to x 2, can be eliminated because it does not contain the minimum. Then, x 2 becomes the new x L for the next round. If f(x 2 )<f(x 1 ), then the domain of x to the right of x 1 from x L to x 2, would have been eliminated. In this case, x 1 becomes the new x u for the next round. Eliminate Minimum

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 31 Eliminate Minimum Old x 2 Old x 1 New x 2 x1x1 xuxu xLxL

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 32 The real benefit from the use of golden ratio is because the original x 1 and x 2 were chosen using golden ratio, we do not need to recalculate all the function values for the next iteration.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example 7.2 Golden Search 33

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 34 x2x2 x1x1 How do we decide whether x 1 or x 2 is our best ‘guess’ for the minimum?

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 35 What would it take to package this into a function?

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parabolic Interpolation Similar to Golden Section The method for improving our guess for the minimum or the maximum is different (In the example shown in the textbook the author used Parabolic interpolation to find a maximum) 36

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Just As… 2 points can be used to determine the equation of a line 3 points can be used to determine the equation for a parabola (a second order polynomial) 37 y=ax+b y=ax 2 + bx + c

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. From the parabolic fit –Differentiate –Set the result equal to 0 –Solve for the root – which corresponds to the max (of the parabola) 38

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display.

Example 7.3 You should be able to create appropriate code to implement parabolic (quadratic) interpolation 40

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. MATLAB Functions for 1-D optimization 41 If you can find the derivative analytically –Use fzero If you can’t find the derivative analytically –Use fminbnd –Uses a combination of the golden section and parabolic techniques To find a maximum just change the sign on the function

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. MATLAB Function for 1-D optimization 42

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Multidimensional Optimization 43

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 45

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 46 Just for fun

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 47

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 49 This is an example of constrained optimization

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 50

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 51 fminsearch uses unconstrained optimization

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Summary We looked at both –Analytical –Numerical techniques to find maximums and minimums for one- dimensional and multi-dimensional problems 52

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Summary Analytical Techniques based on finding the derivative and setting it equal to zero Golden Search Techique Parabolic Technique fzero (which requires you be able to find the analytical derivative – since fzero is a root finding function) fminbnd (Constrained one-dimensional) fminsearch (Unconstrained multi-dimensional) 53