L22 Numerical Methods part 2 Homework Review Alternate Equal Interval Golden Section Summary Test 4 1

2

Problem Yes, descent direction

Prob No, not a descent direction

Prob Slope

Prob cont’d 6

7

Prob

The Search Problem Sub Problem A Which direction to head next? Sub Problem B How far to go in that direction? 9

Search Direction… Min f(x): Let’s go downhill! 10 Descent condition

Step Size? How big should we make alpha? Can we step too “far?” i.e. can our step size be chosen so big that we step over the “minimum?” 11

12 Figure 10.2 Conceptual diagram for iterative steps of an optimization method. We are here Which direction should we head?

Some Step Size Methods “Analytical” Search direction = (-) gradient, (i.e. line search) Form line search function f( α) Find f’( α)=0 Region Elimination (“interval reducing”) Equal interval Alternate equal interval Golden Section 13

14 Figure 10.5 Nonunimodal function f(  ) for 0   Nonunimodal functions Unimodal if stay in locale?

Monotonic Increasing Functions 15

Monotonic Decreasing Functions 16 continous

17 Figure 10.4 Unimodal function f(  ). Unimodal functions monotonic increasing then monotonic decreasing monotonic decreasing then monotonic increasing

Some Step Size Methods “Analytical” Search direction = (-) gradient, (i.e. line search) Form line search function f( α) Find f’( α)=0 Region Elimination (“interval reducing”) Equal interval Alternate equal interval Golden Section 18

19 Figure 10.3 Graph of f(  ) versus . Analytical Step size Slope of line search= Slope of line at fmin

Analytical Step Size Example 20

Alternative Analytical Step Size 21 New gradient must be orthogonal to d for

Some Step Size Methods “Analytical” Search direction = (-) gradient, (i.e. line search) Form line search function f( α) Find f’( α)=0 Region Elimination (“interval reducing”) Equal interval Alternate equal interval Golden Section 22

23 Figure 10.6 Equal-interval search process. (a) Phase I: initial bracketing of minimum. (b) Phase II: reducing the interval of uncertainty. “Interval Reducing” Region elimination “bounding phase” Interval reduction phase”

2 delta! 24

Successive-Equal Interval Algorithm 25 “Interval” of uncertainty

More on bounding phase I Swan’s method Fibonacci sequence 26

Successive Alternate Equal Interval 27 Assume bounding phase has found Min can be on either side of But for sure its not in this region! Point values… not a line

Successive Alt Equal Int 28 Requires two function evaluations per iteration

29 Figure 10.8 Initial bracketing of the minimum point in the golden section method. Fibonacci Bounding

30 Figure 10.9 Graphic of a section partition. Golden section

Golden Section Example 31

Summary General Opt Algorithms have two sub problems: search direction, and step size Descent condition assures correct direction For line searches…in local neighborhood… we can assume unimodal! Step size methods: analytical, region elimin. Region Elimination (“interval reducing”) Equal interval Alternate equal interval Golden Section 32