1. Simulated Annealing & Boltzmann Machines
虞台文
大同大學資工所
智慧型多媒體研究室

2. Content Overview Simulated Annealing Deterministic Annealing
Boltzmann Machines

3. Simulated Annealing & Boltzmann Machines
Overview
大同大學資工所
智慧型多媒體研究室

4. Hill Climbing
E > 0 E
E < 0
E: cost (energy)

5. The Problem with Hill Climbing
Gets stuck at local minima
Gradient decent approach
Hopfield neural networks
Possible solutions
Try different initial states
Increase the size of the neighborhood
(e.g. in TSP try 3-opt rather than 2-opt)

6. Stochastic Approaches
Goal: escape from local-minima.
Stochastic Approaches
Stochastic optimization refers to the minimization (or maximization) of a function in the presence of randomness in the optimization process.
The randomness may be present as either noise in measurements or Monte Carlo randomness in the search procedure, or both.

7. Two Important Methods Simulated Annealing (SA) Genetic Algorithms (GA)
Motivated by the physical annealing process
Evolution from a single solution
Genetic Algorithms (GA)
Motivated by the evolution process of biology
Evolution from multiple solutions

8. Two Important Methods Simulated Annealing (SA) Genetic Algorithms (GA)
Motivated by the physical annealing process
Evolution from a single solution
Genetic Algorithms (GA)
Motivated by the evolution process of biology
Evolution from multiple solutions
Kirkpatrick, S , Gelatt, C.D., Vecchi, M.P “Optimization by Simulated Annealing.”
Science, vol 220, No. 4598, pp
J. Holland, Adaptation in Natural and Artificial Systems, University of Michigan Press, 1975.

9. Simulated Annealing & Boltzmann Machines
大同大學資工所
智慧型多媒體研究室

10. Global Optimization

11. Statistical Mechanics in a Nutshell+ 
Statistical mechanics is the study of the behavior of very large systems of interacting components in thermal equilibrium at a temperature, say T.

12. T Boltzmann Factor kB : Boltzmann constant+ 
kB : Boltzmann constant
Z(T) : Boltzmann partition function

13. T Boltzmann Factor T1 < T2 < T3 E Raising temperature
the system becomes more `active’
the average energy becomes higher
Boltzmann Factor T + 
T1 < T2 < T3 E

14. Simulation Metropolis Acceptance Criterion
E: cost (energy)

15. Simulation Metropolis Acceptance Criterion1
T1 < T2 < T3

16. Simulated Annealing Algorithm
Create initial solution S
Initialize temperature T
repeat
for k = 1 to iteration-length do
Generate a random transition from S to S’
Let E = E(S’)  E(S)
if E < 0 then S = S’
else if exp[E/T] > rand(0,1) then S = S’
Reduce temperature T
until no change in E(S)
Return S

17. Simulated Annealing Algorithm
Hill Climbing
Simulated Annealing Algorithm
Create initial solution S
Initialize temperature T
repeat
for k = 1 to iteration-length do
Generate a random transition from S to S’
Let E = E(S’)  E(S)
if E < 0 then S = S’
else if exp[E/T] > rand(0,1) then S = S’
Reduce temperature T
until no change in E(S)
Return S

18. Main Components of SA Solution representation
Appropriate for computing energy (cost)
Transition mechanism between solutions
Incremental changes of solutions
Cooling schedule
Initial system temperature
Temperature decrement function
Number of iterations between temperature change
Acceptance criteria
Stop criteria

19. Traveling Salesman Problem
Example
Given n-city locations specified in a two-dimensional space, find the minimum tour length. The salesman must visit each and every city only once and should return to the starting city forming a closed path.
Traveling Salesman Problem

20. Traveling Salesman Problem
Example
Traveling Salesman Problem

21. Traveling Salesman Problem
Example
Traveling Salesman Problem

22. Traveling Salesman Problem
Example
Traveling Salesman Problem

23. Traveling Salesman Problem
Example
Traveling Salesman Problem

24. Traveling Salesman Problem
Example
Traveling Salesman Problem

25. Traveling Salesman Problem
Example
Traveling Salesman Problem

26. Traveling Salesman Problem
Example
Traveling Salesman Problem

27. Solution Representation (TSP)
Assume cities are fully connected with symmetric distance. 1 2 3 5 8 9 11 10 6 4 7 ∞

28. Solution Representation (TSP)1 2 3 4 6 5 7 9 11 8 10 1 2 3 5 8 9 11 10 6 4 7
>

29. Energy (Cost) Computation (TSP)
d10,1 1 2 3 4 6 5 7 9 11 8 10
d23
d12
d12
d23
d34
d46
d65
d57
d79
d9.11
d11,8 1 2 3 5 8 9 11 10 6 4 7
d8,10
>
d34
d46
d10,1
d9,11
d65
d11,8
d57
d79
d8,10

30. State Transition (TSP)
d10,1 1 2 3 4 5 6 7 8 9 10
d12
d23
d34
d45
d56
d67
d78
d89
d9,10 1 2 3 4 5 6 7 8 9 10
1. Randomly select two edges

31. State Transition (TSP)
d10,1 1 2 3 4 5 6 7 8 9 10
d34
d89
d12
d23
d45
d56
d67
d78
d9,10 1 2 3 4 5 6 7 8 9 10
1. Randomly select two edges
2. Swap the path

32. State Transition (TSP)
d10,1 1 2 3 4 8 5 7 6 6 7 5 8 4 9 10
d38
d87
d76
d65
d54
d49
d12
d23
d9,10 1 10 2
1. Randomly select two edges 3 9
2. Swap the path 8 4 7 5 6

33. Cooling Schedules Geometric Schedule
Empirical evidence shows that typically 0.8    0.99 yields successful applications (fairly slow cooling schedules).

34. Simulation 100 cities are randomly chosen from 1010 square.
100-city TSP
Simulation

35. Simulated Annealing (T0 is starting temperature)
100 cities are randomly chosen from 1010 square.
100-city TSP
1000N iterations are made for each test.
Simulation
Total
Length
Direct
Search
Simulated Annealing (T0 is starting temperature)
T0=1
T0=2
T0=5
T0=10
T0=25
T0=50
T0=100
Test1
89.211
82.757
82.732
79.113
81.792
82.701
79.405
79.528
Test2
89.755
81.325
81.334
80.532
83.166
80.461
79.25
82.549
Test3
81.44
82.063
84.296
80.629
81.658
80.35
79.21
79.933
Test4
85.038
82.449
82.388
80.996
79.764
82.688
82.131
84.17
Test5
87.256
80.658
80.814
82.261
80.82
81.338
80.406
79.869
Test6
88.989
82.92
81.284
82.607
80.599
83.526
81.43
80.894
Test7
85.895
84.479
80.807
80.402
80.837
79.504
81.052
82.201
Test8
85.654
80.549
81.251
80.195
82.878
80.169
80.604
80.125
Test9
88.246
79.832
79.123
80.617
81.676
80.741
81.306
Test10
84.586
82.446
82.855
81.249
82.885
79.68
81.665
Average
86.607
82.011
81.759
80.711
81.501
81.21
80.476
81.224
Each temperature T is hold for 100N reconfigurations or 10N successful reconfigurations, whichever comes first.
T is reduced by 10% each time.

36. Simulated Annealing (T0 is starting temperature)
100 cities are randomly chosen from 1010 square.
100-city TSP
Simulation
Total
Length
Direct
Search
Simulated Annealing (T0 is starting temperature)
T0=1
T0=2
T0=5
T0=10
T0=25
T0=50
T0=100
Test1
89.211
82.757
82.732
79.113
81.792
82.701
79.405
79.528
Test2
89.755
81.325
81.334
80.532
83.166
80.461
79.25
82.549
Test3
81.44
82.063
84.296
80.629
81.658
80.35
79.21
79.933
Test4
85.038
82.449
82.388
80.996
79.764
82.688
82.131
84.17
Test5
87.256
80.658
80.814
82.261
80.82
81.338
80.406
79.869
Test6
88.989
82.92
81.284
82.607
80.599
83.526
81.43
80.894
Test7
85.895
84.479
80.807
80.402
80.837
79.504
81.052
82.201
Test8
85.654
80.549
81.251
80.195
82.878
80.169
80.604
80.125
Test9
88.246
79.832
79.123
80.617
81.676
80.741
81.306
Test10
84.586
82.446
82.855
81.249
82.885
79.68
81.665
Average
86.607
82.011
81.759
80.711
81.501
81.21
80.476
81.224

37. Simulated Annealing & Boltzmann Machines
Deterministic Annealing
大同大學資工所
智慧型多媒體研究室

38. The Problems of SA
SA techniques are inherently slow because of their randomized local search strategy.
Converge to global optimum in probability one sense only if the cooling schedule is in the order of

39. The Problems of SA
Geman, S. & Geman, D. (1984) “Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images,” IEEE Trans. on Pattern Analysis and Machine Intelligence 6,
SA techniques are inherently slow because of their randomized local search strategy.
Converge to global optimum in probability one sense only if the cooling schedule is in the order of
Geman and Geman [1984]

40. Review Simulated Annealing Algorithm
Create initial solution S{0, 1}n
Initialize temperature T
repeat
for k = 1 to iteration-length do
Generate a random transition from S to S’ by inverting a random bit si
Let E = E(S’)  E(S)
if E < 0 then S = S’
else if exp[E/T] > rand(0,1) then S = S’
Reduce temperature T
until no change in E(S)
Return S

41. Review Simulated Annealing Algorithm
Create initial solution S{0, 1}n
Initialize temperature T
repeat
for k = 1 to iteration-length do
Generate a random transition from S to S’ by inverting a random bit si
Let E = E(S’)  E(S)
if E < 0 then S = S’
else if exp[E/T] > rand(0,1) then S = S’
Reduce temperature T
until no change in E(S)
Return S
Stochastic nature

42. Deterministic Annealing (DA)
Also called mean-field annealing.
Deterministic Annealing (DA)
Create initial solution S[0, 1]n
Initialize temperature T
repeat
for k = 1 to iteration-length do
Choose a random bit si
Reduce temperature T
until convergence criterion met
Return S
T1 < T2 < T3 1
Deterministic behavior

43. Simulated Annealing & Boltzmann Machines
大同大學資工所
智慧型多媒體研究室

44. + Boltzmann Machines Discrete Hopfield NN Boltzmann Machine Simulated
Annealing

45. Update Rules
Discrete Hopfield NN
Boltzmann Machine
Unipolar neuron

46. Update Rules Discrete Hopfield NN Boltzmann Machine Unipolar neuron
Cooling schedule is required.
Update Rules
T=0
T=1
T=2
T=3
T=
Discrete Hopfield NN
Boltzmann Machine
Unipolar neuron

47. Exercises
Computer Simulations on the same TSP problem demonstrated previously using
Simulated Annealing
Deterministic Annealing, and
Boltzmann Machine.
Perform some analyses on your results.