2. Artificial Intelligence Search Methodologies Dr Rong Qu School of Computer Science University of Nottingham Nottingham, NG8 1BB, UK rxq@cs.nott.ac.uk Evolutionary Computation

3. Based on the idea of evolution EAs - framework Dr. R Qu G54STA/G54SDA – Evolutionary Computing PopulationParents Offsprings Metaheuristics – From Design to Implementation pp. 199 (Talbi, 2009) Selection Reproduction Replacement MetaphorOptimization EvolutionProblem solving IndividualSolution FitnessObjective function EnvironmentOptimization problem

4. Template of an evolutionary algorithm EAs - framework Dr. R Qu G54STA/G54SDA – Evolutionary Computing Algorithm. Template of an evolutionary algorithm Generate(P(0)); //initial population t = 0; While not Termination_Criterion(P(t)) Do Evaluate(P(t)); P’(t) = Selection(P(t)); P’(t) = Reproduction(P’(t)); Evaluate(P’(t)); P(t+1) = Replace(P(t), P’(t)); t = t+1; End While Output Best individual or best population found

5. Genetic Algorithms Charles Darwin 1809 - 1882 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

6. Genetic Algorithms 1859 Origin of the Species Survival of the Fittest Dr. R Qu G54STA/G54SDA – Evolutionary Computing

7. Genetic Algorithms 1975 Genetic Algorithms Artificial Survival of the Fittest Dr. R Qu G54STA/G54SDA – Evolutionary Computing

8. Genetic Algorithms 1989 Genetic Algorithms Foundations and Applications Dr. R Qu G54STA/G54SDA – Evolutionary Computing

9. Compared to exhaustive tree search techniques, modern heuristic algorithms quickly converge to sub- optimal solutions by searching a small fraction of the search space Simulates the Darwinian evolution process in computer programs. Populations evolve by selective pressures, mating between individuals, and alterations such as mutations. Genetic operators evolve solutions in the current population to create a new population Genetic Algorithms Dr. R Qu G54STA/G54SDA – Evolutionary Computing

10. Generate Initial Population Generate Initial Population Generation 'n' Population Generation 'n' Crossover Population Crossover Population Mutate Population Mutate Population n = n + 1 n = 1 Final Population Final Population Genetic Algorithms Flowchart (20 generations) Selection n<20? Dr. R Qu G54STA/G54SDA – Evolutionary Computing

11.  Evolution of population (a set of solutions) into a new population, using operators: reproduction, mutation and crossover  Can be implemented as three modules: the evaluation, the population and the reproduction modules Genetic Algorithms Dr. R Qu G54STA/G54SDA – Evolutionary Computing

12. Crossover Genetic Algorithm Operators A percentage of the population is selected for breeding and assigned random mates. A random crossover point is selected and a pair of new solutions are produced 1 2 1 2 Adapted from Prof Graham Kendall’s G5BAIM lecture example

13. Mutation Genetic Algorithm Operators In a small percentage of the population random changes are made. May be designed to conduct more complex operations (local search) accordingly. Dr. R Qu G54STA/G54SDA – Evolutionary Computing

14. Genetic Algorithm Example I Maximise f(x) = x^3 - 60 * x^2 + 900 * x +100 0 <= x <= 31 Crossover probability, PC = 1.0 Mutation probability, PM = 0.0 x can be represented using five binary digits Adapted from Dr Graham Kendall’s G5BAIM lecture example Dr. R Qu G54STA/G54SDA – Evolutionary Computing

15. Genetic Algorithm Example I chromosomebinary stringxf(x) P11110028212 P201111153475 P310111231227 P40010042804 Total7718 Average1929.50 Maximise f(x) = x^3 - 60 * x^2 + 900 * x +100 0 <= x <= 31 Generating random initial population Dr. R Qu G54STA/G54SDA – Evolutionary Computing

16. Genetic Algorithm Example I Roulette wheelParents 1227P3 3475P2 10111 01111 P3P3 P2P2 11111 00111 C1C1 C2C2 00100 01111 P4P4 P2P2 00111 01100 C3C3 C4C4 Maximise f(x) = x^3 - 60 * x^2 + 900 * x +100 0 <= x <= 31 Choose two parents (roulette selection) One point crossover (random position 1) Dr. R Qu G54STA/G54SDA – Evolutionary Computing

17. Genetic Algorithm Example I chromosomebinary stringxf(x) P11111131131 P20011173803 P30011173803 P401100123889 Total11735 Average2933.75 Maximise f(x) = x^3 - 60 * x^2 + 900 * x +100 0 <= x <= 31 New generation Dr. R Qu G54STA/G54SDA – Evolutionary Computing

18. Genetic Algorithm Example I chromosomebinary stringxf(x) P11110028212 P201111153475 P310111231227 P40010042804 Total7718 Average1929.50 chromosomebinary stringxf(x) P11111131131 P20011173803 P30011173803 P401100123889 Total11735 Average2933.75 Maximise f(x) = x^3 - 60 * x^2 + 900 * x +100 0 <= x <= 31 Two generations Mutation What problem do you see with the populations? Dr. R Qu G54STA/G54SDA – Evolutionary Computing

19. Genetic Algorithm Example II Traveling Salesman Problem (TSP) a number of cities costs of traveling between cities a traveling sales man needs to visit all these cities exactly once and return to the starting city What’s the cheapest route? Dr. R Qu G54STA/G54SDA – Evolutionary Computing

20. Genetic Algorithm Example II Traveling Salesman Problem Dr. R Qu G54STA/G54SDA – Evolutionary Computing

21. Genetic Algorithm Example II Initial generation 581……8432275467 P1P1 788127……91174424 P2P2 817……916362419 P 30 … 6.5 7.8 6.0 Any idea of other ways to generate the initial population? Dr. R Qu G54STA/G54SDA – Evolutionary Computing

22. Genetic Algorithm Example II Choose pairs of parents 788127……91174424 P2P2 817……916362419 P 30 6.0 7.8 Crossover 788127……916362419 C2C2 817……91174424 C1C1 5.9 6.2 13 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

23. Genetic Algorithm Example II Next generation 788127……916362419 P2P2 817……91174424 P1P1 5.9 6.2 782……51076479 P 30 6.0 … Dr. R Qu G54STA/G54SDA – Evolutionary Computing

24. Genetic Algorithm Example II Traveling Salesman Problem No. of cities: 100 Population size: 30 Cost: 6.37 Generation: 88 Cost: 6.34 Generation: 1100 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

25. Applying Genetic Algorithms to Personnel Scheduling Applying Genetic Algorithms to Personnel Scheduling Genetic Algorithm Example III Personnel scheduling in healthcare is usually a very complex operation which has a profound effect upon the efficient usage of expensive resources. E.K. Burke, T. Curtois, R. Qu and G. Vanden Berghe. "A Time Predefined Variable Depth Search for Nurse Rostering". INFORMS Journal on Computing, 2012. doi: 10.1287/ijoc.1120.0510 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

26. A number of nurses A number of shifts each day A set of constraints Genetic Algorithm Example III shift coverage one shift per day resting time workload per month consecutive shifts working weekends …

27. 26 Dr. R Qu G54SAI – Evolutionary Algorithms Genetic Algorithm - Initial population - construct rosters - repair infeasible ones Genetic Algorithm Example III

28. Genetic Algorithm - Select parents - Recombine rows in the two rosters - repair infeasible ones + Genetic Algorithm Example III

29. 28 Dr. R Qu G54SAI – Evolutionary Algorithms Genetic Algorithm - Mutation - Local optimiser Genetic Algorithm Example III

30. Population Size Crossover Probability Mutation Probability Local Optimiser 50 0.7 0.001 ON Genetic Algorithm Example III Dr. R Qu G54STA/G54SDA – Evolutionary Computing

31. GA - performance There are a number of factors which affect the performance of a genetic algorithm The size of the population The initial population Selection pressure (elitism, tournament) The cross-over probability The mutation probability Defining convergence Local optimisation Dr. R Qu G54STA/G54SDA – Evolutionary Computing

32. Survival of the fittest, the most fit of a particular generation (the best possible answer) are used as parents for the next generation. GAs are a very powerful tool, allowing searching for solutions when there are no other feasible means to do so. More advanced evolutionary algorithms emerged in the last two decades. GA - Conclusions Dr. R Qu G54STA/G54SDA – Evolutionary Computing

33. Evaluation is responsible for evaluating the fitness of individuals in the population It is the only part of the GA that has any knowledge about the specific problem. The rest of the GA modules are simply operating on chromosomes with no information about the problem. A different evaluation module is needed for each problem. Dr. R Qu G54STA/G54SDA – Evolutionary Computing Evolutionary Algorithms

34. Conceivable solutions are represented, the 'fittest' will have the greatest chance of reproducing. Successful properties will therefore be favourably represented within the population of solutions. The population is successively 'optimised' after a number of generations. Dr. R Qu G54STA/G54SDA – Evolutionary Computing Evolutionary Algorithms

35. Representation Population Objective function Selection Replacement Stopping Dr. R Qu G54STA/G54SDA – Evolutionary Computing Evolutionary Algorithms

36. Estimation of Distribution Algorithms (EDA) Dr. R Qu G54STA/G54SDA – Evolutionary Computing

37. Selection of better individuals Initial Population Generation n = 1 Initial Population Generation n = 1 Build probability model n = n + 1 Final Population Final Population EDA Flowchart (20 generations) n<20? Sample model to generate new individuals New Population Dr. R Qu G54STA/G54SDA – Evolutionary Computing

38. A recent class of population based metaheuristics Uses probability distribution of the population Statistical information extracted from the population used to generate the next population Probability model (vector) Replaces the genetic operators to evolve the population EDA Dr. R Qu G54STA/G54SDA – Evolutionary Computing

39. Template of an EDA Adapted from Metaheuristics – From Design to Implementation pp. 223 (Talbi, 2009) Algorithm. Template of an EDA t = 1; Generate randomly a population of n individuals Initialize a probability model Q(x); While not Termination_Criterion Do Create P(t) a population of n individuals by sampling from Q(x); Evaluate(P(t)); Select m individuals from P(t) using a selection method; Update the probability model Q(x) using the selected individuals; t = t+1; End While Output Best individual or best population found Dr. R Qu G54STA/G54SDA – Evolutionary Computing

40. Template of a PBIL Adapted from Metaheuristics – From Design to Implementation pp. 224 (Talbi, 2009) Algorithm. Template of a PBIL algorithm Initial distribution D = (0.5, 0.5, …, 0.5); Repeat // Generate the population For each individual Xi If r < Dj (r uniform in [0,1]) Then xij = 1 Else xij = 0; End For Evaluate and sort the population; Update the distribution D = (1-α)D + αX best ; Until Stopping criteria Dr. R Qu G54STA/G54SDA – Evolutionary Computing

41. Maximise f(x) = x^3 - 60 * x^2 + 900 * x +100 0 <= x <= 31 x represented using 5 binary digits Probability Vector 0.5 PBIL Example I Dr. R Qu G54STA/G54SDA – Evolutionary Computing

42. PBIL Example I individualsbinary stringXf(X) P11110028212 P201111153475 P310111231227 P40010042804 Average1929.50 Maximise f(x) = x^3 - 60 * x^2 + 900 * x +100 0 <= x <= 31 Generating initial population based on Evaluate the population Probability Vector 0.5 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

43. PBIL Example I Maximise f(x) = x^3 - 60 * x^2 + 900 * x +100 0 <= x <= 31 Update the PV based on Xbest individualsbinary stringXf(X) P11110028212 P201111153475 P310111231227 P40010042804 Average1929.50 Probability Vector 0.450.65 D = (1-α)D + αX best Probability Vector 0.5 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

44. individualsbinary stringxf(x) P101011114071 P20011173803 P301101133857 P401110113684 Average3853.75 Maximise f(x) = x^3 - 60 * x^2 + 900 * x +100 0 <= x <= 31 Generating new population based on Probability Vector 0.450.65 PBIL Example I Mutation in GA vs. Randomness in PBIL Probability Vector …………… D = (1-α)D + αX best Dr. R Qu G54STA/G54SDA – Evolutionary Computing

45. 44 PBIL Example II Portfolio Selection Problem Given a set of assets (stocks, etc.) and their associated return and risk What’s the best investment plan (i.e. which assets to choose to achieve the highest return and lowest risk)? Mathworks

46. 45 PBIL Example II Portfolio Selection Problem Modern Portfolio Theory Nobel memorial prize winner Harry Markowitz Diversification: “don’t put all your eggs into one basket” Which assets to choose? How much to invest to each asset? Real world constraints Asset Allocation and Portfolio Optimization, Mathworks

47. PBIL Example II Portfolio Selection Problem Which assets to choose? A binary vector of size n n: number of assets How much to invest to each asset? A real vector of size n, n: number of assets Additional constraints Cardinality: a limited number of assets Boundary: a lower and upper bound of investment to the assets K. Lwin, R. Qu. "Hybrid Algorithm for Constrained Portfolio Selection Problem". Applied Intelligence, 39(2): 251-266, 2013. doi: 10.1007/s10489-012-0411-7 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

48. PBIL Example II Portfolio Selection Problem Evaluation function Return: combined return based on historical data Risk: combined risk (variance of return) of all chosen assets Constraint handling Cardinality Boundaries Dr. R Qu G54STA/G54SDA – Evolutionary Computing

49. Combinatorial optimisation problems bin packing problems vehicle routing problems job shop scheduling Evolutionary Algorithm Applications

50. Combinatorial optimisation problems portfolio optimization multimedia multicast routing knapsack problem Evolutionary Algorithm Applications

51. APPENDIX Other Evolutionary Algorithms Dr. R Qu G54STA/G54SDA – Evolutionary Computing

52. ANT ALGORITHMS Ants are practically blind but they still manage to find their way to and from food. How do they do it? Dr. R Qu G54STA/G54SDA – Evolutionary Computing

53. Ant Algorithms Ant systems are a population based approach. In this respect it is similar to genetic algorithms There is a population of ants, with each ant finding a solution and then communicating with the other ants Dr. R Qu G54STA/G54SDA – Evolutionary Computing

54. Ant Algorithms A B C H D F E G Dr. R Qu G54STA/G54SDA – Evolutionary Computing

55. Ant Algorithms A B C D F E d=0.5 d=1 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

56. Ant Algorithms Time, t, is discrete At each time unit an ant moves a distance, d of 1 Once an ant moved, it lays down 1 unit of pheromone At t = 0, there is no pheromone on any edge Dr. R Qu G54STA/G54SDA – Evolutionary Computing

57. Ant Algorithms At t=1 there will be 16 ants at B and 16 ants at D. At t=2 there will be 8 ants at D and 8 ants at B. There will be 16 ants at E The intensities on the edges will be as follows FD = 16, AB = 16, BE = 8, ED = 8, BC = 16 and CD = 16 A B C D F E 0.5 1 1 1 1 16 ants are moving from A - F and another 16 are moving from F - A Dr. R Qu G54STA/G54SDA – Evolutionary Computing

58. Ant Algorithms We need to allow the ants to explore paths and follow the best paths with some probability in proportion to the intensity of the pheromone trail We do not want them simply to follow the route with the highest amount of pheromone on it, else our search will quickly settle on a sub-optimal (and probably very sub-optimal) solution Dr. R Qu G54STA/G54SDA – Evolutionary Computing

59. Ant Algorithms The probability of an ant following a certain route is a function, not only of the pheromone intensity but also a function of what the ant can see (visibility) The pheromone trail must not build unbounded. Therefore, we need “evaporation” Dr. R Qu G54STA/G54SDA – Evolutionary Computing

60. Ant Algorithms – initial ideas Dorigo (1996) Based on real world phenomena Ants, despite almost blind, are able to find their way to the food source using the shortest route If an obstacle is placed, ants have to decide which way to take around the obstacle. Dr. R Qu G54STA/G54SDA – Evolutionary Computing

61. Ant Algorithms – initial ideas Dorigo (1996) Initially there is a 50-50 probability as to which way they will turn Assume one route is shorter than the other Ants taking the shorter route will arrive at a point on the other side of the obstacle before the ants which take the longer route. Dr. R Qu G54STA/G54SDA – Evolutionary Computing

62. Ant Algorithms – initial ideas Dorigo (1996) As ants walk they deposit pheromone trail. Ants have taken shorter route will have already laid trail So ants from the other direction are more likely to follow that route with deposit of pheromone. Dr. R Qu G54STA/G54SDA – Evolutionary Computing

63. Ant Algorithms – initial ideas Dorigo (1996) Over a period of time, the shortest route will have high levels of pheromone. The quantity of pheromones accumulates faster on the shorter path than on the longer one There is positive feedback which reinforces that behaviors so that the more ants follow a particular route, the more desirable it becomes. Dr. R Qu G54STA/G54SDA – Evolutionary Computing

64. Ant Algorithms Dr. R Qu G54STA/G54SDA – Evolutionary Computing

65. Ant Algorithms - TSP At the start of the algorithm, one ant is placed in each city Time, t, is discrete. t(0) marks the start of the algorithm. At t+1 every ant will have moved to a new city Assuming that the TSP is represented as a fully connected graph, each edge has an intensity of trail on it. This represents the pheromone trail laid by the ants Let T i,j (t) represent the intensity of trail edge (i,j) at time t Variations have been tested by Dorigo Dr. R Qu G54STA/G54SDA – Evolutionary Computing

66. Ant Algorithms - TSP An ant decides which town to move to next, with a probability that is based on the distance to that city AND the amount of trail intensity on the connecting edge The distance to the next town, is known as the visibility, n ij, defined as 1/d ij, d ij is the distance between cities i and j. At each time unit evaporation takes place, at a rate p, a value between 0 and 1 Variations have been tested by Dorigo Dr. R Qu G54STA/G54SDA – Evolutionary Computing

67. Ant Algorithms - TSP In order to stop ants visiting the same city in the same tour a data structure, Tabu, is maintained Tabu k is defined as the list for the k th ant and it holds the cities that have already been visited Variations have been tested by Dorigo Dr. R Qu G54STA/G54SDA – Evolutionary Computing

68. Ant Algorithms - TSP After each ant tour the trail intensity on edge (i,j) is updated using the following formula T ij (t + n) = p. T ij (t) + ΔT ij Q is a constant L k is the tour length of the k th ant p is the evaporation coefficient Dr. R Qu G54STA/G54SDA – Evolutionary Computing

69. Ant Algorithms - TSP Transition Probability where  and  are control parameters that control the relative importance of trail versus visibility Dr. R Qu G54STA/G54SDA – Evolutionary Computing

70. Ant Algorithms - TSP Left: Trail distribution at the beginning; Right: Trail distribution after 100 cycles. (Dorigo et al., 1996) Dr. R Qu G54STA/G54SDA – Evolutionary Computing

71. Ant Algorithms - Applications Travelling Salesman Problem (TSP) Facility Layout Problem Vehicle Routing Stock Cutting … Marco Dorigo maintains a page devoted to the subject at http://iridia.ulb.ac.be/~mdorigo/ACO/ACO.html contains information about ant algorithms as well as links to the main papers published on the subject Dr. R Qu G54STA/G54SDA – Evolutionary Computing