Instructor: Prof.Dr.Sahand Daneshvar Presented by: Seyed Iman Taheri Student number: Non linear Optimization Spring EASTERN MEDITERRANEAN UNIVERSITY Department of Industrial Engineering

Line Search Using Derivatives bisection search method Newton's method

bisection search method Suppose that we wish to minimize a function θ over a closed and bounded interval. Furthermore, suppose that θ is pseudoconvex and hence, differentiable. At iteration k, let the interval of uncertainty be [a k, b k ]. Suppose that the derivative θ'(λ k ) is known, and consider the following three possible cases:

1. If θ'(λ k ) = 0, then, by the pseudoconvexity of θ, λ k is a minimizing point. 2. If θ'(λ k ) > 0, then, for λ > λ k we have θ'(λ k ) (λ- λ k ) > 0; and by the pseudoconvexity of θ it follows that θ(λ) ≥θ(λ k ). In other words, the minimum occurs to the left of λ k, so that the new interval of uncertainty [a k+l, b k+1 ]is given by [a k, λ k ]. 3. If θ'(λ k ) 0, so that θ(λ) ≥θ(λ k ). Thus, the minimum occurs to the right of λ k, so that the new interval of uncertainty [a k+l, b k+1 ] is given by [λ k, b k ].

The position of λ k, in the interval [a k, b k ] must be chosen so that the maximum possible length of the new interval of uncertainty is minimized. That is, λ k must be chosen so as to minimize the maximum of λ k - a k and b k - λ k. Obviously, the optimal location of λ k is the midpoint (1/2)(a k + b k ). To summarize, at any iteration k, θ' is evaluated at the midpoint of the interval of uncertainty. Based on the value of θ', we either stop or construct a new interval of uncertainty whose length is half that at the previous iteration. Note that this procedure is very similar to the dichotomous search method except that at each iteration, only one derivative evaluation is required, as opposed to two functional evaluations for the dichotomous search method. However, the latter is akin to a finite difference derivative evaluation.

Convergence of the Bisection Search Method Note that the length of the interval of uncertainty after n observations is equal to (1/2) n (b 1 -a 1 ), so that the method converges to a minimum point within any desired degree of accuracy. In particular, if the length of the final interval of uncertainty is fixed at C, then n must be chosen to be the smallest integer such that (1/2) n ≤ l /(b 1 -a 1 ).

Summary of the Bisection Search Method We now summarize the bisection search procedure for minimizing a pseudoconvex function θ over a closed and bounded interval. Initialization Step Let [a 1, b 1 ] be the initial interval of uncertainty, and let P be the allowable final interval of uncertainty. Let n be the smallest positive integer such that (1/2) n ≤ l /(b 1 -a 1 ). Let k = 1 and go to the Main Step. Main Step Let λ k = (1/2)(a k + b k ) and evaluate θ'(λ k ). If θ'(λ k )=0, stop; λ k is an optimal solution. Otherwise, go to Step 2 if θ'(λ k )> 0, and go to Step 3 if θ'(λ k )< Let a k+1 = a k and b k+1 = λ k. GO to Step Let a k+1 = λ k, and b k+1 = b k. GO to Step If k = n, stop; the minimum lies in the interval [ a n+1, b n+1 ]. Otherwise, replace k by k + 1 and repeat Step 1.

Example Consider the following problem: Minimize (λ) 2 + 2λ subject to -3≤λ≤6. Suppose that we want to reduce the interval of uncertainty to an interval whose length e is less than or equal to 0.2. Hence, the number of observations n satisfying (1/2) n ≤ l /(b 1 -a 1 )= 0.2/9 = is given by n=6. A summary of the computations using the bisection search method is given in Table of next slide.

Summary of Computations for the Bisection Search Method Note that the final interval of uncertainty is [ , ], so that the minimum could be taken as the midpoint,

Newton’s Method Newton’s method is based on exploiting the quadratic approximation of the function θ at a given point λ k. This quadratic approximation q is given by q(λ) = θ(λ k ) + θ΄ (λ k )(λ - λ k ) + (1/2)θ"̋(λ k )(λ - λ k ) 2. The point λk+1, is taken to be the point where the derivative of q is equal to zero. This yields θ΄ (λ k )+ θ"̋(λ k )(λ k+1 - λ k ) = 0, so that λ k+1 = λ k –(θ΄ (λ k )/θ"̋(λ k )) The procedure is terminated when |λk+1 - λk| < ε, or when |θ΄ (λ k )|< ε, where ε is a pre-specified termination scalar. Note that the above procedure can only be applied for twice differentiable functions. Furthermore, the procedure is well defined only if θ"̋(λ k )≠0 for each k.

Example Consider the function θ: Note that θ is twice differentiable everywhere. We apply Newton’s method, starting from two different points. In the first case, λ 1 = 0.40; and as shown in first Table of next slide, the procedure produces the point after six iterations. The reader can verify that the procedure indeed converges to the stationary point λ =0. In the second case, λ= 0.60, and the procedure oscillates between the points 0.60 and -0.60, as shown in second Table of next slide.

Summary of Computations for Newton's Method Starting from λ1 = 0.4 Summary of Computations for Newton‘s Method Starting from λ 1 = 0.6

Convergence of Newton's Method The method of Newton, in general, does not converge to a stationary point starting with an arbitrary initial point. Observe that, in general, Theorem cannot be applied as a result of the unavailability of a descent function. However, as shown in next Theorem, if the starting point is sufficiently close to a stationary point, then a suitable descent function can be devised so that the method converges.

Theorem

Proof

Thanks for your attention