1. Method of Hooke and Jeeves

2. Multidimensional Search Without Using Derivatives
In this section we consider the problem of minimizing a function f of several variables without using derivatives. The methods described here proceed in the following manner.

3. Multidimensional Search Without Using Derivatives
Given a vector x, a suitable direction d is first determined, and then f is minimized from x in the direction d by one of the techniques discussed earlier. X = ( x1 , x2 , … , xn ) Direction = dx Minimizing F(x) in dx direction

4. What is the problem?
We are required to solve a line search problem of the form to: minimize f(x + λd) subject to: λ L where L is typically of the form: L = { λ : λ ≥ 0 } or L = R L = { λ : a ≤ λ ≤ b }

5. What is the problem?
In the statements of the algorithms, for the purpose of simplicity we have assumed that a minimizing point exists. However, this may not be the case. Here, the optimal objective value of the line search problem may be:
Unbounded: Then the original problem is unbounded and we may stop.
Finite but not achieved at any particular λ: Then λ could be chosen as such that
f(x+ d) is sufficiently close to the value:
inf { f(x+λd): λ L }.

6. Method of Hooke and Jeeves
The method of Hooke and Jeeves performs two types of search:
Exploratory search
Pattern search

7. Method of Hooke and Jeeves
Given x1, an exploratory search along the coordinate directions produces the point x2. Now a pattern search along the direction x2- x1 leads to the point y. y
Pattern Search x2
Exploratory search along
the coordinate axes x1

8. Method of Hooke and Jeeves
Another exploratory search starting from y gives the point x3. The next pattern search is conducted along the direction x3 –x2, yielding y'. The process is then repeated. y
Pattern Search y' x2 x3 x1
Exploratory search along
the coordinate axes

9. Summary of the Method of Hooke and Jeeves Using Line Searches
As originally proposed by Hooke and Jeeves, the method does not perform any line search but rather takes discrete steps along the search directions, as we discuss later. Here we present a continuous version of the method using line searches along the coordinate directions d1 ,..., dn and the pattern direction.

10. Initialization Step
Choose a scalar ε > 0 to be used in terminating the algorithm.
Choose a starting point x1
let y1 = x1
let k = j = 1
Go to the Main Step.

11. Main Step Step 1: minimize f(yj +λdj) subject to: λ R yj+1 = yj +λj dj
Let λj be an optimal solution to the problem to:
minimize f(yj +λdj)
subject to: λ R
yj+1 = yj +λj dj
If j < n, replace j by j + 1, and repeat Step 1. Otherwise, if j = n, let xk+1=yn+1 .
If ||xk+1- xk|| < ε , stop;
otherwise, go to Step2.

12. Main Step Step 2: minimize f(xk+1+ λd) Subject to: λ R
Let d = xk+1 – xk
Let be an optimal solution to the problem to:
minimize f(xk+1+ λd)
Subject to: λ R
Let y1 = xk+1+ d
Let j = 1
replace k by k+1
Go to Step 1.

13. Example I Consider the following problem:
Minimize (x1 – 2)4 +(x1 – 2x2)2
Note that the optimal solution is (2, 1) with objective value equal to zero.
At each iteration:
An exploratory search along the coordinate directions gives the points y2 and y3
A pattern search along the direction d = xk+1 – xk gives the point y1
Except at iteration k = 1, where y1 = x1
Four iterations were required to move from the initial point to the optimal point (2, 1) whose objective value is zero.
At this point, ||x5 – x4|| = 0.045
and the procedure is terminated.

14. Summary of Computations for the Method of Hooke and Jeeves Using Line Searches

15. Method of Hooke and Jeeves using line searches.

16. Point
The Pattern Search has substantially improved the Convergence behavior by moving along a direction that is almost parallel to the valley shown by dashed lines.

17. Convergence of the Method of Hooke and Jeeves
Suppose that:
f differentiable
Let the solution set Ω= { : f ( )= 0).
Note: Each iteration of the method of Hooke and Jeeves consists of an application of the cyclic coordinate method, in addition to a pattern search.
Let the cyclic coordinate search be denoted by the map B and the pattern search be denoted by the map C.
If the minimum of f along any line is unique and letting
α = f
then α(y) < α( x ) for x Ω.
By the definition of C, α(z) ≤ α(y) for z C(y).
Assuming that Λ = { x : f ( x ) < f ( x1 ) } , where x1 is the starting point, is compact, convergence of the procedure is established.

18. Method of Hooke and Jeeves with Discrete Steps
The method of Hooke and Jeeves, as originally proposed, does not perform line searches but, instead, adopts a simple scheme involving functional evaluations. A summary of the method is given below:

19. Initialization Step Let d1,...,dn be the coordinate directions.
Choose a scalar ε > 0 to be used for terminating the algorithm.
choose an initial step size, Δ ≥ ε.
Choose an acceleration factor,α>0.
Choose a starting point x1, let y1= x1, let k=j=1
Go to the Main Step.

20. Main Step Comparing f(yj+Δdj) and f(yj):
If f(yj+Δdj) < f(yj), the trial is termed a success; let yj+1= yj+Δdj , and go to Step2.
If f(yj+Δdj) ≥ f(yj), the trial is deemed a failure. In this case:
If f(yj -Δdj) < f(yj), let yj+1= yj -Δdj , and go to Step 2;
if f(yj -Δdj) ≥ f(yj), let yj+1= yj, and go to Step 2 .

21. Main Step Comparing j and n Do as follows:
If j < n, replace j by j+1, and repeat Step 1.
Otherwise:
if f(yn+1) < f(xk) go to Step 3.
if f(yn+1) ≥ f(xk) go to Step 4.
Do as follows:
Let xk+1 = yn+1
Let y1 = xk+1 + α(xk+1 – xk)
Replace k by k+ l
Let j = 1
Go to Step 1.

22. Main Step Compare Δ and ε
If Δ ≤ ε, stop ; xk is the prescribed solution.
Otherwise, replace Δ by Δ /2.
Let y1 = xk
Let xk+1=xk
Replace k by k+ l ,
Let j = 1
Repeat Step 1.

23. Note Steps I and II describe an exploratory search.
Step III is an acceleration step along the direction xk+1- xk.
A decision whether to accept or reject the acceleration step is not made until after an exploratory search is performed.
In Step IV, the step size Δ is reduced. The procedure could easily be modified so that different step sizes are used along the different directions. This is sometimes adopted for the purpose of scaling.

24. Consider the following problem: Minimize (x1 – 2)4 + (x1- 2 x2 )2
Example II: Solve the problem using the method of Hooke and Jeeves with discrete steps
Consider the following problem:
Minimize (x1 – 2)4 + (x1- 2 x2 )2
In which the parameters α and Δ are chosen as 1.0 and 0.2, respectively.
The algorithm start from (0.0, 3.0).
The points generated are numbered equentially.
The acceleration step that is rejected is shown by the dashed lines.

25. Method of Hooke and Jeeves using discrete steps starting from (0. 0,3
Method of Hooke and Jeeves using discrete steps starting from (0.0,3.0). The numbers denote the order in which points are generated.

26. (S) denotes that the trial is a Success
The following table give a more comprehensive illustration, It summarizes the computations starting from the new initial point (2.0, 3.0). Here:
(S) denotes that the trial is a Success
(F) denotes that the trial is a Failure
At the iteration, and at subsequent iterations whenever:
If f(y3) ≥ f (xk) then the vector y1 is taken as xk .
Otherwise, y1 = 2xk+1- xk .
Note: At the end iteration k =10, the point (1.70, 0.80) is reached having an objective value 0.02.
The procedure is stopped here with the termination parameter ε = 0.1.
If a greater degree of accuracy is required, Δ should be reduced to 0.05.

27. The Summary Table of Computations for the Method of Hooke and Jeeves with Discrete Steps

29. Method of Hooke and Jeeves using line searches
Method of Hooke and Jeeves using line searches. The numbers denote the order in which Donets are generated.

30. Thank you all for your attention