Advanced Computer Graphics Optimization Part 2 Spring 2002 Professor Brogan

Simulated Annealing Review of Metropolis Procedure –Evaluation function –Current state –Temperature (and annealing schedule) –Change in state

Computing New States Lacking derivatives (and when data isn’t smooth) –Doesn’t matter With derivatives –If you know which way is downhill (better) Then why risk losing that improvement by picking a new direction randomly

Going Downhill If you always use derivative to go downhill –Metropolis step is ineffective (you’re always going to take the better answer) Process turns into a simple gradient descent algorithm

Gradient Descent Follow derivatives downhill Performs poorly in narrow valleys And performs worse as the solution is reached

Best of Both Worlds Current state = Simplex of N+1 points –Not just one point Envision simplex with lines connecting N+1 points –In three dimensions it is a tetrahedron The goal is for minimum to be spanned by points At each iteration, replace one point with a lower-valued one

Simplex Method Diameter of simplex gets smaller with each iteration Stop when diameter reaches tolerance

Example Find minimum of f(x,y) Simplex is a triangle –(N+1 = 3) Start with –A = (x 1, y 1 ), B = (x 2, y 2 ), C = (x 3, y 3 ) Reflection –Compute a new point, D = (x 4, y 4 ) that is reflection of highest point (A in this example) through midpoint of other two points D = B + C - A

Example Expansion –If D is smaller than A –Then move was good –Try going further in same direction –E = 2D – (B + C) / 2

Example Contraction –If D has same value as A, then find F and G F = (2A + B + C) / 4 G = (2D +B + C) / 4 –The smallest of F and G is kept

Example Shrinkage If neither F nor G are smaller than A Then the side connecting A and C must move towards B in order to shrink the simplex –H = (A + B) / 2 –I = (B + C) / 2

Back to Simulated Annealing Remember our state is N+1 points We add a positive, logarithmically distributed random variable, proportional to the temperature, T, to each point in state We subtract a similar random variable from the function value of each new point that is tried as a replacement point

Simulated Annealing As temp goes to 0, this becomes Simplex At other T values –Brownian motion of simplex shape and sampling new, random points

Want to read more? Numerical Recipes in C mlhttp:// ml Source code can be found online too

Golden Search in 1D The root of a function (where it equals 0) can be found when two points bracket it –One point has value > 0 –Other point has value < 0 –Somewhere in middle, function has value = 0

Finding Min is Different 3 points are needed to find min –f(a) > f(b) –f(c) > f(b) –Min is certainly between a and c Similar to root finding algorithms like Bisection method –Choose a new point x that is between a and b or b and c

Updating Points Try a point x between b and c If f(x) < f(b) –Points = (b, x, c) Else –Points = (a, b, x) Continue until distance between outer points is small

Tolerance What is small enough span? (1-e)b < b < (1+e)b –Where e = 3x10 -8 (float) or 3x (double) –Shape of f(x) near b is given by Taylor series –Thus, second term is tiny compared to the first and will act just like 0 when added to it –So keep e large (1x10 -4 (float))

Golden Mean Where should new intermediate point be for (a, b, c)? Let b be fraction w from between a and c Suppose the next trial point were fraction z beyond b: (x – b) / (c – a) Next segment will either be of length w+z or 1- w To minimize worst case possibility, select z so these two are equal: z = 1 – 2w

Golden Mean Resulting point is –|b-a| = |x-c| z will be in second portion if w < ½ Because w was actually selected using this same algorithm in previous step –x (at dist z) should be same fraction of way from b to c (if that was bigger seg) as was b from a to c (=w) –z / (1-w) = w Solve for w = (Golden mean)