1. 02 -1 Lecture 02 Heuristic Search Topics –Basics –Hill Climbing –Simulated Annealing –Best First Search –Genetic Algorithms –Game Search

2. 02 -2 Basics Heuristics –General rules about past experience of solving a problem Problem solving by heuristic search –Solving a problem by reasoning about how to search a solution space using heuristics Solution space –A space containing (partial) solution states Search –Exploring solution spaces by generating solution states

3. 02 -3 Hill Climbing Search solution spaces using gradient as heuristics Always search the direction with the greatest gradient B A C A Current state: A => Next state: B

4. 02 -4 Hill Climbing Solving 8-puzzle by hill climbing using Manhattan distance heuristics –Manhattan distance 123 456 78 467 12 538 S goal =((1 2 3) (4 5 6) (7 8 #)) S initial =((4 6 7) (1 # 2) (5 3 8)) MD(Ts1) = ||(2, 1), (1, 1)|| = |2-1| + |1-1| = 1 − Heuristics

5. 02 -5 Hill Climbing Solving 8-puzzle by hill climbing using Manhattan distance heuristics S initial =((4 6 7)(1 # 2)(5 3 8)) [H=-16] ((4 # 7)(1 6 2)(5 3 8)) [H=-15] ((# 4 7)(1 6 2)(5 3 8)) [H=-17] ((4 6 7)(# 1 2)(5 3 8)) [H=-17] ((4 7 #)(1 6 2)(5 3 8)) [H=-14] ((4 6 7)(1 3 2)(5 # 8)) [H=-15] ((4 6 7)(1 2 #)(5 3 8)) [H=-15] O#左O#左 O#右O#右 O#左O#左 O#右O#右 O#下O#下 O#上O#上 O#下O#下

6. 02 -6 Hill Climbing Hill-climbing(S initial ) { S current ← S initial ; H(S current ) ← Heuristics-eval(S current ); loop { if ?Goal(S current ) = true, exit; New-states ← Legal-operators(S current ); if New-states = , exit; for each S i  New-states H(S i ) ← Heuristics-eval(S i ); S max ← ； if H(S max ) ≦ H(S current ), exit; S current ← S max ; } return S current ; }

7. 02 -7 Hill Climbing Problems with hill climbing –Local maxima Plateau Ridge Hill top

8. 02 -8 Simulated Annealing Annealing refers to the process of cooling down a high temperature liquidated metal or glass back to a solid state in order to get the solid material settled in an optimal state. We have to carefully control the temperature during annealing so that at any time point the system is approximately under thermodynamic equilibrium, which implies equilibrium entropy is reached Given equilibrium entropy, a system contains maximal possible states according to the second law of thermodynamics –Boltzmann defined entropy [S] as log W where W is the number of microstates in the system time a constant "k" ; hence S= k Log W

9. 02 -9 Simulated Annealing The probability of each (energy) state E at some temperature T can be determined by Boltzmann Factor p(E) : k: Boltzmann constant P(E 2 ) P(E 1 ) P(E 3 ) E 1 < E 2 < E 3 Energy Prob Fixed temperature

10. 02 -10 Simulated Annealing Annealing schedule: a mechanism that controls annealing temperature; it includes initial annealing temperature as well as how the temperature is decreased Simulated annealing: –Simulate the annealing process to explore all possible states in terms of state probability to reach the optimal final state

11. 02 -11 Simulated Annealing Simulated annealing uses the Monte Carlo Method to simulate the annealing process –For a given state E, random() simulates the minimal probability that the state is promising –If the Boltzmann Factor of state E satisfies the following condition, it is considered to be a choice for further exploration: Simulated annealing provides a mechanism to escape from local maxima Simulated annealing does not guarantee to final optimal solutions though Annealing schedule –Initial annealing temperature: T 0 –Temperature change : T i+1 =  T i

12. 02 -12 Simulated Annealing B D A B D A C (b) (a) Current state: B => Next state: C if Compared to Hill Climbing

13. 02 -13 Simulated Annealing Simulated-annealing (S initial ){ S current ← S initial ; H(S current ) ← Heuristics-eval(S current ); T next ← Annealing-schedule(); loop{ New-states ← Legal-operators(S current ); if New-states = , exit; for each S i  New-states H(S i ) ← Heuristics-eval(S i ); S new ← ; if (H(S new ) > H(S current )) then S current ← S new ; else loop { S new ← Random-select(New-states); If Boltzmann-factor(S new, T next ) > Random(1), then {S current ← S new ; exit;} } if ?End-annealing-schedule() = true, exit; else T next = Annealing-schedule(); } return S current ; }

14. 02 -14 Best First Search Go global to cope with local maxima Use priority queue Q p to contain all nodes waiting for exploration Q p represents the wave front (the frontier of the exploration process) Always choose the best state from Q p to explore next Completeness of Best First Search –can always find optimal solutions

15. 02 -15 Best First Search Solving 8-puzzle using Best First Search S initial =((4 6 7)(1 # 2)(5 3 8)) [H=-16] ((4 # 7)(1 6 2)(5 3 8)) [H=-15] ((# 4 7)(1 6 2)(5 3 8)) [H=-17] ((4 6 7)(# 1 2)(5 3 8)) [H=-17] ((4 7 #)(1 6 2)(5 3 8))[H=-14] ((4 6 7)(1 3 2)(5 # 8)) [H=-15] ((4 6 7)(1 2 #)(5 3 8)) [H=-15] O#左O#左 O#右O#右 O#左O#左 O#右O#右 O#上O#上 O#下O#下 O#下O#下

16. 02 -16 Best First Search Current state: B => Next state: D, if, after B is explored, D becomes the best in Q p B D A QpQp B D A C QpQp (b) (a) Compared to Hill Climbing

17. 02 -17 Best First Search Best-first-search (S initial ) { H(S initial ) ← Heuristics-eval(S initial ); Add-pqueue(Q p, S initial, H(S initial )); loop{ S max ← Pop-pqueue(Q p ) ； if S max = nil, fail ； else S current ← S max ; if ?Goal(S current ) = true, exit; New-states ← Legal-operators(S current ); if New-states = , exit; for each S i  New-states { H(S i ) ← Heuristics-eval(S i ); Add-pqueue(Q p, S i, H(S i )); } return S current ; }

18. 02 -18 Best First Search A* algorithm –A Best First Search process with a heuristic which insists the estimated cost must never be larger than the real optimal cost –That is, h(S) ≦ h*(S), given heuristics evaluation function f(S)=g(S)+h(S), where h(S) and h*(S) represent the estimated and optimal cost, respectively, from the current state to the target state Optimality of A* –Can always find optimal solutions through optimal paths

19. 02 -19 Genetic Algorithms Simulate natural evolution process –The fittest survives –Chromosomes vs. solutions –Genetic operators vs. search –Fitness value vs. objective function Genetic operators –Selection –Crossover –Mutation

20. 02 -20 Genetic Algorithms Selection –Selection strategy Elitism Variety –Elitism Selection probabilities proportional to fitness values Crossover –Crossover rate –Crossover point Mutation –Single- vs. multiple-point mutation –Mutation rate

21. 02 -21 Genetic Algorithms Example Initial population selection –Begin with a collection of initial solutions –Solution example: –Transform each solution into a bit string X = 155 Y = 124 Z = 228 ( 155, 124, 228) ( 100110110111110011100100) XYZ

22. 02 -22 Genetic Algorithms Selection –Select two parents from the population according to a cumulated fitness distribution formed as a roulette wheel ( 155, 124, 228) ( 100110110111110011100100) X Y Z ( 116, 4, 195) ( 011101000000010011000011)

23. 02 -23 Genetic Algorithms

24. 02 -24 Genetic Algorithms Crossover –Determine whether the two selected individuals are qualified for crossover (crossover rate) –Identify locations for crossovers –Crossovers need not be at gene boundaries –Exchange the trailing portions of the bit string to create two new children (155, 124, 228)(100110110111110011100100) (116, 4, 195)(011101000000010011000011) (100110000000110011100111) (011101110111010011000000)(119, 116, 192) (152, 12, 231)

25. 02 -25 Genetic Algorithms Mutation –Mutation points Point mutation –select a random bit in the string and change it Complex mutation –mutate a pattern or sequence of bits –Determine whether the selected bit deserves mutation (mutation rate) (152, 12, 231)(100110000000110011100111) (152, 76, 231)(100110000100110011100111)

26. 02 -26 Genetic Algorithms Genetic Algorithm(P, Y, N, L){ /* P pop., Y gen., N chromo., L long */ P current ← P; gen ← 1; f old ← Average(Fitness-value(P current )); loop until gen>Y { PD ← Selection-prob(P current ); RW ← Produce-cum-prob(PD); P new ←  ; i ← 1; loop until i>N { d 1 ← Select(P current, RW); d 2 ← Select(P current, RW); ← Crossover(d1, d2, p c, L); d m1 ← Mutate(d c1, p m, L ); Add(d m1, P new ); d m2 ← Mutate(d c2, p m, L ); Add(d m2, P new ); i = i +2; } P current ← P new ; f new ← Average(Fitness-value(P current )); if |f new - f old | ≦ , exit; else { f old ← f new ; gen = gen + 1; } return P current and the fittest chromosome in P current ; }

27. 02 -27 Genetic Algorithms Select(P, RW) { i ← Roulette-wheel(RW, P, Random()) ； Return i ； } Crossover(d 1, d 2, p c, L) { if p c >Random() { b ← Random(L) ； d c1 ← Substring(d 1, 1, b) + Substring(d 2, b+1, L) ； d c2 ← Substring(d 2, 1, b) + Substring(d 1, b+1, L) ； } else ← ； return ； } Mutate(d, p m, L) { b ← 1 ； loop until b>L { if p m > Random() change the b th bit of d; b = b + 1 ； } return d; }

28. 02 -28 Genetic Algorithms Why Genetic algorithm works to find optimal solutions? –Schema theorem - Schema with higher fitness value, shorter definition length, and less definition genes get exponential growth during the evolution process Higher fitness value: higher selection probability Shorter defining length: less crossover probability Less defining genes: less mutation probability

29. 02 -29 Genetic Algorithms Gene position: 12345678 Chromosome schema ： **0**1** chromosomes included ： 01010110, 010010110, 00010110, … Defining length ： 6 -3 = 3 (position of 0 is 3 and that of 1 is 6) Defining genes ： 2 (0 and 1)

30. 02 -30 Game Search Adversarial search MiniMax Strategy –Select maximally evaluated values from my moves –Select minimally evaluated values from opponent’s moves Terminal (look ahead 2 steps) A 246128 1452 B C D 3 32 2 3 MAX MIN Max(3, 2, 2) Min(3, 12, 8)

31. 02 -31 MiniMax on Tic-tac-toe Game Search

32. 02 -32 Game Search MiniMax on Tic- tac-toe (cont.)

33. 02 -33 MiniMax Algorithm 1. Generate the game tree down to the terminal nodes. 2. Apply the utility function to the terminal nodes. 3. For a S set of sibling nodes, pass up to the parent… 1) the lowest value in S if the siblings are 2) the largest value in S if the siblings are 4. Recursively do the above, until the backed-up values reach the initial state. 5. The value of the initial state is the minimum score for Max. Game Search

34. 02 -34 Game Search Alpha-Beta Pruning –In general, if m is better than n, the subtree leading to n should never be explored A B m Player Opponent n Player Opponent …..

35. 02 -35 Game Search  -Cutoff ≥2 2 2 Player (MAX) Opponent (MIN) ≤1 78 1

36. 02 -36 Game Search  -cutoff ≥2 5 ≤5 Player (MAX) Opponent (MIN) ≥7 2 57 Player (MAX)