1. Downhill Simplex Search (Nelder-Mead Method)
J.-S. Roger Jang (張智星)
MIR Lab, CSIE Dept.
National Taiwan University
2019/4/11

2. Downhill Simplex Search (DSS)
DSS: A heuristic search method that tries to find the minimizing point in a function of several variables
Proposed by John Nelder & Roger Mead in 1965
AKA amoeba method, Nelder-Mead method
Concept
Use a simplex of n+1 points to explore the objective function with n inputs
Simplex: interval in 1D, triangle in 2D, tetrahedron in 3D, etc.
Simplex have several operations to move to and pinpoint the minimum: reflection, expansion, contraction, shrink
Quiz!

3. Basic Operation: Reflection
Reflection toward downhill direction
location probed by reflection step
worst point (highest function value)

4. Basic Operation: Expansion
Try an expansion if reflection resulted in lowest value
location probed by expansion step

5. Basic Operation: Contraction
Try a contraction if reflection resulted in highest value
location probed by contration step

6. Basic Operation: Shrink
Shrink the simplex around the best point if all else fails.

7. DSS Parameters for DSS R=1 (reflection) K=0.5 (contraction)
E=2 (expansion)
S=0.5 (shrinkage)

8. Steps in DSS (1/4)
Sort by function value: Order the vertices to satisfy f1 < f2 < … < fn+1
Calculate xm = sum xi (the average of all the points except the worst)
Reflection. Compute xr = xm + a(xm-xn+1) and evaluate fr =f(xr). If f1 < fr < fn , accept xr and terminate the iteration.

9. Steps in DSS (2/4)
Expansion. If fr < f1 calculate xe = xm+ g(xr - xm) and evaluate fe =f(xe). If fe < fr , accept xe ; otherwise accept xr. Terminate the iteration.

10. Steps in DSS (3/4)
Contraction. If fr > fn, perform a contraction between xm and the better of xr and xn+1.
Outside. If fn < fr < fn+1 calculate xoc= xm+ l (xr - xm) and evaluate f(xoc). If foc< fr, accept xoc and terminate the iteration; otherwise do a shrink.
Inside. If fr > fn+1 calculate xic = xm – K (xm- xn+1) and evaluate f(xic). If fic< fn+1 accept xic and terminate the iteration; otherwise do a shrink.

11. Steps in DSS (4/4)
Shrink. Evaluate f at the n points vi = xi + S (xi-x1), i = 2,….,n+1. The vertices of the simplex at the next iteration are x1, v2, …, vn+1.

12. List of DSS Moves in 2D

13. Example of DSS (1/2)

14. Example of DSS (2/2)

15. MATLAB for Optimization
Basic functions for optimization
Single variable: fminbnd
Multiple variables: fminsearch (which uses DSS)
Examples of using the above functions
請見「MATLAB 程式設計進階篇」的第8.4節「函數的極小值」
Slides and recording
For more options, try the following toolboxes
Optimization Toolbox
Global Optimization Toolbox

16. Summary of DSS About DSS Strengths Weaknesses Hints
Intuitive concept
Easy to implement
No gradient or derivative needed
Does not care about smoothness of the objective function
Weaknesses
Slow
Could be trapped in local minima
Hints
Start with various initial guesses to avoid local minima.
Improvise your own version of DSS!
Quiz!

17. References DSS in Wikipedia Some of our slides come from
Many variants!
Some of our slides come from