Downhill Simplex Search (Nelder-Mead Method) J.-S. Roger Jang (張智星) jang@mirlab.org http://mirlab.org/jang MIR Lab, CSIE Dept. National Taiwan University 2019/4/11

Downhill Simplex Search (DSS) DSS: A heuristic search method that tries to find the minimizing point in a function of several variables Proposed by John Nelder & Roger Mead in 1965 AKA amoeba method, Nelder-Mead method Concept Use a simplex of n+1 points to explore the objective function with n inputs Simplex: interval in 1D, triangle in 2D, tetrahedron in 3D, etc. Simplex have several operations to move to and pinpoint the minimum: reflection, expansion, contraction, shrink Quiz!

Basic Operation: Reflection Reflection toward downhill direction location probed by reflection step worst point (highest function value)

Basic Operation: Expansion Try an expansion if reflection resulted in lowest value location probed by expansion step

Basic Operation: Contraction Try a contraction if reflection resulted in highest value location probed by contration step

Basic Operation: Shrink Shrink the simplex around the best point if all else fails.

DSS Parameters for DSS R=1 (reflection) K=0.5 (contraction) E=2 (expansion) S=0.5 (shrinkage)

Steps in DSS (1/4) Sort by function value: Order the vertices to satisfy f1 < f2 < … < fn+1 Calculate xm = sum xi (the average of all the points except the worst) Reflection. Compute xr = xm + a(xm-xn+1) and evaluate fr =f(xr). If f1 < fr < fn , accept xr and terminate the iteration.

Steps in DSS (2/4) Expansion. If fr < f1 calculate xe = xm+ g(xr - xm) and evaluate fe =f(xe). If fe < fr , accept xe ; otherwise accept xr. Terminate the iteration.

Steps in DSS (3/4) Contraction. If fr > fn, perform a contraction between xm and the better of xr and xn+1. Outside. If fn < fr < fn+1 calculate xoc= xm+ l (xr - xm) and evaluate f(xoc). If foc< fr, accept xoc and terminate the iteration; otherwise do a shrink. Inside. If fr > fn+1 calculate xic = xm – K (xm- xn+1) and evaluate f(xic). If fic< fn+1 accept xic and terminate the iteration; otherwise do a shrink.

Steps in DSS (4/4) Shrink. Evaluate f at the n points vi = xi + S (xi-x1), i = 2,….,n+1. The vertices of the simplex at the next iteration are x1, v2, …, vn+1.

List of DSS Moves in 2D

Example of DSS (1/2)

Example of DSS (2/2)

MATLAB for Optimization Basic functions for optimization Single variable: fminbnd Multiple variables: fminsearch (which uses DSS) Examples of using the above functions 請見「MATLAB 程式設計進階篇」的第8.4節「函數的極小值」 Slides and recording For more options, try the following toolboxes Optimization Toolbox Global Optimization Toolbox

Summary of DSS About DSS Strengths Weaknesses Hints Intuitive concept Easy to implement No gradient or derivative needed Does not care about smoothness of the objective function Weaknesses Slow Could be trapped in local minima Hints Start with various initial guesses to avoid local minima. Improvise your own version of DSS! Quiz!

References DSS in Wikipedia Some of our slides come from Many variants! Some of our slides come from http://physics.ujep.cz/~mmaly/vyuka/MPVT_II/Heuristiky/NelderMead.ppt http://www.cs.princeton.edu/courses/archive/spr06/cos323/notes/lecture03_optimization/cos323_s06_lecture03_optimization.ppt