Artificial Intelligence Search Methodologies Dr Rong Qu School of Computer Science University of Nottingham Nottingham, NG8 1BB, UK Evolutionary Computation

Based on the idea of evolution EAs - framework Dr. R Qu G54STA/G54SDA – Evolutionary Computing PopulationParents Offsprings Metaheuristics – From Design to Implementation pp. 199 (Talbi, 2009) Selection Reproduction Replacement MetaphorOptimization EvolutionProblem solving IndividualSolution FitnessObjective function EnvironmentOptimization problem

Template of an evolutionary algorithm EAs - framework Dr. R Qu G54STA/G54SDA – Evolutionary Computing Algorithm. Template of an evolutionary algorithm Generate(P(0)); //initial population t = 0; While not Termination_Criterion(P(t)) Do Evaluate(P(t)); P’(t) = Selection(P(t)); P’(t) = Reproduction(P’(t)); Evaluate(P’(t)); P(t+1) = Replace(P(t), P’(t)); t = t+1; End While Output Best individual or best population found

Genetic Algorithms Charles Darwin Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithms 1859 Origin of the Species Survival of the Fittest Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithms 1975 Genetic Algorithms Artificial Survival of the Fittest Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithms 1989 Genetic Algorithms Foundations and Applications Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Compared to exhaustive tree search techniques, modern heuristic algorithms quickly converge to sub- optimal solutions by searching a small fraction of the search space Simulates the Darwinian evolution process in computer programs. Populations evolve by selective pressures, mating between individuals, and alterations such as mutations. Genetic operators evolve solutions in the current population to create a new population Genetic Algorithms Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Generate Initial Population Generate Initial Population Generation 'n' Population Generation 'n' Crossover Population Crossover Population Mutate Population Mutate Population n = n + 1 n = 1 Final Population Final Population Genetic Algorithms Flowchart (20 generations) Selection n<20? Dr. R Qu G54STA/G54SDA – Evolutionary Computing

 Evolution of population (a set of solutions) into a new population, using operators: reproduction, mutation and crossover  Can be implemented as three modules: the evaluation, the population and the reproduction modules Genetic Algorithms Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Crossover Genetic Algorithm Operators A percentage of the population is selected for breeding and assigned random mates. A random crossover point is selected and a pair of new solutions are produced Adapted from Prof Graham Kendall’s G5BAIM lecture example

Mutation Genetic Algorithm Operators In a small percentage of the population random changes are made. May be designed to conduct more complex operations (local search) accordingly. Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithm Example I Maximise f(x) = x^ * x^ * x <= x <= 31 Crossover probability, PC = 1.0 Mutation probability, PM = 0.0 x can be represented using five binary digits Adapted from Dr Graham Kendall’s G5BAIM lecture example Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithm Example I chromosomebinary stringxf(x) P P P P Total7718 Average Maximise f(x) = x^ * x^ * x <= x <= 31 Generating random initial population Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithm Example I Roulette wheelParents 1227P3 3475P P3P3 P2P C1C1 C2C P4P4 P2P C3C3 C4C4 Maximise f(x) = x^ * x^ * x <= x <= 31 Choose two parents (roulette selection) One point crossover (random position 1) Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithm Example I chromosomebinary stringxf(x) P P P P Total11735 Average Maximise f(x) = x^ * x^ * x <= x <= 31 New generation Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithm Example I chromosomebinary stringxf(x) P P P P Total7718 Average chromosomebinary stringxf(x) P P P P Total11735 Average Maximise f(x) = x^ * x^ * x <= x <= 31 Two generations Mutation What problem do you see with the populations? Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithm Example II Traveling Salesman Problem (TSP) a number of cities costs of traveling between cities a traveling sales man needs to visit all these cities exactly once and return to the starting city What’s the cheapest route? Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithm Example II Traveling Salesman Problem Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithm Example II Initial generation 581…… P1P …… P2P2 817…… P 30 … Any idea of other ways to generate the initial population? Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithm Example II Choose pairs of parents …… P2P2 817…… P Crossover …… C2C2 817…… C1C Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithm Example II Next generation …… P2P2 817…… P1P …… P … Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Genetic Algorithm Example II Traveling Salesman Problem No. of cities: 100 Population size: 30 Cost: 6.37 Generation: 88 Cost: 6.34 Generation: 1100 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Applying Genetic Algorithms to Personnel Scheduling Applying Genetic Algorithms to Personnel Scheduling Genetic Algorithm Example III Personnel scheduling in healthcare is usually a very complex operation which has a profound effect upon the efficient usage of expensive resources. E.K. Burke, T. Curtois, R. Qu and G. Vanden Berghe. "A Time Predefined Variable Depth Search for Nurse Rostering". INFORMS Journal on Computing, doi: /ijoc Dr. R Qu G54STA/G54SDA – Evolutionary Computing

A number of nurses A number of shifts each day A set of constraints Genetic Algorithm Example III shift coverage one shift per day resting time workload per month consecutive shifts working weekends …

26 Dr. R Qu G54SAI – Evolutionary Algorithms Genetic Algorithm - Initial population - construct rosters - repair infeasible ones Genetic Algorithm Example III

Genetic Algorithm - Select parents - Recombine rows in the two rosters - repair infeasible ones + Genetic Algorithm Example III

28 Dr. R Qu G54SAI – Evolutionary Algorithms Genetic Algorithm - Mutation - Local optimiser Genetic Algorithm Example III

Population Size Crossover Probability Mutation Probability Local Optimiser ON Genetic Algorithm Example III Dr. R Qu G54STA/G54SDA – Evolutionary Computing

GA - performance There are a number of factors which affect the performance of a genetic algorithm The size of the population The initial population Selection pressure (elitism, tournament) The cross-over probability The mutation probability Defining convergence Local optimisation Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Survival of the fittest, the most fit of a particular generation (the best possible answer) are used as parents for the next generation. GAs are a very powerful tool, allowing searching for solutions when there are no other feasible means to do so. More advanced evolutionary algorithms emerged in the last two decades. GA - Conclusions Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Evaluation is responsible for evaluating the fitness of individuals in the population It is the only part of the GA that has any knowledge about the specific problem. The rest of the GA modules are simply operating on chromosomes with no information about the problem. A different evaluation module is needed for each problem. Dr. R Qu G54STA/G54SDA – Evolutionary Computing Evolutionary Algorithms

Conceivable solutions are represented, the 'fittest' will have the greatest chance of reproducing. Successful properties will therefore be favourably represented within the population of solutions. The population is successively 'optimised' after a number of generations. Dr. R Qu G54STA/G54SDA – Evolutionary Computing Evolutionary Algorithms

Representation Population Objective function Selection Replacement Stopping Dr. R Qu G54STA/G54SDA – Evolutionary Computing Evolutionary Algorithms

Estimation of Distribution Algorithms (EDA) Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Selection of better individuals Initial Population Generation n = 1 Initial Population Generation n = 1 Build probability model n = n + 1 Final Population Final Population EDA Flowchart (20 generations) n<20? Sample model to generate new individuals New Population Dr. R Qu G54STA/G54SDA – Evolutionary Computing

A recent class of population based metaheuristics Uses probability distribution of the population Statistical information extracted from the population used to generate the next population Probability model (vector) Replaces the genetic operators to evolve the population EDA Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Template of an EDA Adapted from Metaheuristics – From Design to Implementation pp. 223 (Talbi, 2009) Algorithm. Template of an EDA t = 1; Generate randomly a population of n individuals Initialize a probability model Q(x); While not Termination_Criterion Do Create P(t) a population of n individuals by sampling from Q(x); Evaluate(P(t)); Select m individuals from P(t) using a selection method; Update the probability model Q(x) using the selected individuals; t = t+1; End While Output Best individual or best population found Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Template of a PBIL Adapted from Metaheuristics – From Design to Implementation pp. 224 (Talbi, 2009) Algorithm. Template of a PBIL algorithm Initial distribution D = (0.5, 0.5, …, 0.5); Repeat // Generate the population For each individual Xi If r < Dj (r uniform in [0,1]) Then xij = 1 Else xij = 0; End For Evaluate and sort the population; Update the distribution D = (1-α)D + αX best ; Until Stopping criteria Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Maximise f(x) = x^ * x^ * x <= x <= 31 x represented using 5 binary digits Probability Vector 0.5 PBIL Example I Dr. R Qu G54STA/G54SDA – Evolutionary Computing

PBIL Example I individualsbinary stringXf(X) P P P P Average Maximise f(x) = x^ * x^ * x <= x <= 31 Generating initial population based on Evaluate the population Probability Vector 0.5 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

PBIL Example I Maximise f(x) = x^ * x^ * x <= x <= 31 Update the PV based on Xbest individualsbinary stringXf(X) P P P P Average Probability Vector D = (1-α)D + αX best Probability Vector 0.5 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

individualsbinary stringxf(x) P P P P Average Maximise f(x) = x^ * x^ * x <= x <= 31 Generating new population based on Probability Vector PBIL Example I Mutation in GA vs. Randomness in PBIL Probability Vector …………… D = (1-α)D + αX best Dr. R Qu G54STA/G54SDA – Evolutionary Computing

44 PBIL Example II Portfolio Selection Problem Given a set of assets (stocks, etc.) and their associated return and risk What’s the best investment plan (i.e. which assets to choose to achieve the highest return and lowest risk)? Mathworks

45 PBIL Example II Portfolio Selection Problem Modern Portfolio Theory Nobel memorial prize winner Harry Markowitz Diversification: “don’t put all your eggs into one basket” Which assets to choose? How much to invest to each asset? Real world constraints Asset Allocation and Portfolio Optimization, Mathworks

PBIL Example II Portfolio Selection Problem Which assets to choose? A binary vector of size n n: number of assets How much to invest to each asset? A real vector of size n, n: number of assets Additional constraints Cardinality: a limited number of assets Boundary: a lower and upper bound of investment to the assets K. Lwin, R. Qu. "Hybrid Algorithm for Constrained Portfolio Selection Problem". Applied Intelligence, 39(2): , doi: /s Dr. R Qu G54STA/G54SDA – Evolutionary Computing

PBIL Example II Portfolio Selection Problem Evaluation function Return: combined return based on historical data Risk: combined risk (variance of return) of all chosen assets Constraint handling Cardinality Boundaries Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Combinatorial optimisation problems bin packing problems vehicle routing problems job shop scheduling Evolutionary Algorithm Applications

Combinatorial optimisation problems portfolio optimization multimedia multicast routing knapsack problem Evolutionary Algorithm Applications

APPENDIX Other Evolutionary Algorithms Dr. R Qu G54STA/G54SDA – Evolutionary Computing

ANT ALGORITHMS Ants are practically blind but they still manage to find their way to and from food. How do they do it? Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms Ant systems are a population based approach. In this respect it is similar to genetic algorithms There is a population of ants, with each ant finding a solution and then communicating with the other ants Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms A B C H D F E G Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms A B C D F E d=0.5 d=1 Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms Time, t, is discrete At each time unit an ant moves a distance, d of 1 Once an ant moved, it lays down 1 unit of pheromone At t = 0, there is no pheromone on any edge Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms At t=1 there will be 16 ants at B and 16 ants at D. At t=2 there will be 8 ants at D and 8 ants at B. There will be 16 ants at E The intensities on the edges will be as follows FD = 16, AB = 16, BE = 8, ED = 8, BC = 16 and CD = 16 A B C D F E ants are moving from A - F and another 16 are moving from F - A Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms We need to allow the ants to explore paths and follow the best paths with some probability in proportion to the intensity of the pheromone trail We do not want them simply to follow the route with the highest amount of pheromone on it, else our search will quickly settle on a sub-optimal (and probably very sub-optimal) solution Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms The probability of an ant following a certain route is a function, not only of the pheromone intensity but also a function of what the ant can see (visibility) The pheromone trail must not build unbounded. Therefore, we need “evaporation” Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms – initial ideas Dorigo (1996) Based on real world phenomena Ants, despite almost blind, are able to find their way to the food source using the shortest route If an obstacle is placed, ants have to decide which way to take around the obstacle. Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms – initial ideas Dorigo (1996) Initially there is a probability as to which way they will turn Assume one route is shorter than the other Ants taking the shorter route will arrive at a point on the other side of the obstacle before the ants which take the longer route. Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms – initial ideas Dorigo (1996) As ants walk they deposit pheromone trail. Ants have taken shorter route will have already laid trail So ants from the other direction are more likely to follow that route with deposit of pheromone. Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms – initial ideas Dorigo (1996) Over a period of time, the shortest route will have high levels of pheromone. The quantity of pheromones accumulates faster on the shorter path than on the longer one There is positive feedback which reinforces that behaviors so that the more ants follow a particular route, the more desirable it becomes. Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms - TSP At the start of the algorithm, one ant is placed in each city Time, t, is discrete. t(0) marks the start of the algorithm. At t+1 every ant will have moved to a new city Assuming that the TSP is represented as a fully connected graph, each edge has an intensity of trail on it. This represents the pheromone trail laid by the ants Let T i,j (t) represent the intensity of trail edge (i,j) at time t Variations have been tested by Dorigo Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms - TSP An ant decides which town to move to next, with a probability that is based on the distance to that city AND the amount of trail intensity on the connecting edge The distance to the next town, is known as the visibility, n ij, defined as 1/d ij, d ij is the distance between cities i and j. At each time unit evaporation takes place, at a rate p, a value between 0 and 1 Variations have been tested by Dorigo Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms - TSP In order to stop ants visiting the same city in the same tour a data structure, Tabu, is maintained Tabu k is defined as the list for the k th ant and it holds the cities that have already been visited Variations have been tested by Dorigo Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms - TSP After each ant tour the trail intensity on edge (i,j) is updated using the following formula T ij (t + n) = p. T ij (t) + ΔT ij Q is a constant L k is the tour length of the k th ant p is the evaporation coefficient Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms - TSP Transition Probability where  and  are control parameters that control the relative importance of trail versus visibility Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms - TSP Left: Trail distribution at the beginning; Right: Trail distribution after 100 cycles. (Dorigo et al., 1996) Dr. R Qu G54STA/G54SDA – Evolutionary Computing

Ant Algorithms - Applications Travelling Salesman Problem (TSP) Facility Layout Problem Vehicle Routing Stock Cutting … Marco Dorigo maintains a page devoted to the subject at contains information about ant algorithms as well as links to the main papers published on the subject Dr. R Qu G54STA/G54SDA – Evolutionary Computing