Evolutionary Computation: A Tutorial September 2009 Biointelligence Laboratory School of Computer Sci. & Eng. Seoul National University

Contents Part I: Introduction to Evolutionary Computation Part II: Genetic Programming Part III: Advanced Topics Part IV: Applications Part V: Summary and Further Info (C) SNU CSE Biointelligence Lab2

Part I: Introduction to Evolutionary Computation

Overview The Basic Concept General Framework of Evolutionary Computation Components of Evolutionary Computation Paradigms in Evolutionary Computation 4(C) SNU CSE Biointelligence Lab

Basic Concept Use of Darwinian-like evolutionary processes to solve difficult computational problems.  Hence the name, “Evolutionary Computation” Biological basis  Biological systems adapt themselves to a new environment by evolution.  Generations of descendants are produced that perform better than do their ancestors. Biological evolution  Production of descendants changed from their parents  Selective survival of some of these descendants to produce more descendants 5(C) SNU CSE Biointelligence Lab

Basic Concept What is the Evolutionary Computation?  Stochastic search (or problem solving) techniques that mimic the metaphor of natural biological evolution. Metaphor EVOLUTION Individual Fitness Environment PROBLEM SOLVING Candidate Solution Quality Problem 6

(C) SNU CSE Biointelligence Lab General Framework of EC Generate Initial Population Evaluate Fitness Select Parents Generate New Offspring Termination Condition? Yes No Fitness Function Crossover, Mutation Best Individual 7

Components of EC (1/9) Representations Population Fitness function Selection mechanism Variation operators  Crossover/Mutation Initialization / Termination 8(C) SNU CSE Biointelligence Lab

Components of EC (2/9) An example: the 8 queens problem  Place 8 queens on an 8x8 chessboard in such a way that they cannot check each other. 9

Components of EC (3/9) Representations  How to represent the space to be searched?  Genotype  Internal representation of solutions in EC.  Minimum domain knowledge.  Phenotype  External representation of solutions for the problem.  Require domain knowledge. Be careful so that every feasible solution can be represented in genotype space. 10(C) SNU CSE Biointelligence Lab

Genotype: a permutation of the numbers Phenotype: a board configuration Obvious mapping 11 Representation Example: 8 Queens Problem

Components of EC (4/9) Population  Usually has a fixed size and is a multiset of genotypes  Some sophisticated EAs also assert a spatial structure on the population e.g., a grid.  Diversity of a population refers to the number of different fitnesses / phenotypes / genotypes present (note not the same thing). Population sizing  Parent population size (M), offspring population size (K).  Typically M = K.  In typical ES (explained later), M << K.  In steady state algorithms, M > K. 12(C) SNU CSE Biointelligence Lab

Components of EC (5/9) Fitness function  Represents the requirements that the population should adapt to  a.k.a. quality function or objective function  Assigns a single real-valued fitness to each phenotype which forms the basis for selection  So the more discrimination (different values) the better  Typically we talk about fitness being maximised  Some problems may be best posed as minimisation problems, but conversion is trivial. 13

Fitness Function Example: 8 Queens Problem Penalty for a queen:  The number of queens she can check Penalty for a configuration:  The sum of penalties of all queens Note: penalty is to be minimized Fitness of a configuration:  inverse penalty to be maximized (C) SNU CSE Biointelligence Lab

Components of EC (6/9) Parent selection mechanism  Assigns probabilities for an individual to be selected as parents.  Selection probabilities are relative to current population.  Different probabilities can be assigned to the same individuals.  Usually depends on the individual’s fitness and probabilistic.  High quality solutions more likely to become parents than low quality ones  Selection pressure – the degree of correlation between the individual’s fitness and its selection probability.  High selection pressure results in reducing search scope.  Even worst in current population usually has non-zero probability of becoming a parent  This stochastic nature can aid escape from local optima. 15

Selection Mechanism Example: 8 Queens Problem /9 = /10 = 0.1 1/11 = /6 = 0.17 IndividualProportionalRanking (1) (2) (3) (4) (1) (2) (3) (4) Different selection mechanisms assign different selection probabilities for the same individual. 16(C) SNU CSE Biointelligence Lab

Components of EC (7/9) Crossover (Recombination)  Mix information from parents into offspring in stochastic way.  Most offspring may be worse, or the same as the parents.  Hope is that some are better by combining elements of genotypes that lead to good traits.  Principle has been used for millennia by breeders of plants and livestock Example

(C) SNU CSE Biointelligence Lab Components of EC (8/9) Mutation  It is applied to one genotype and delivers a (slightly) modified mutant, the child or offspring of it.  Element of randomness is essential.  The role of mutation in EC is different in various EC subtypes. Example  swapping values of two randomly chosen positions

(C) SNU CSE Biointelligence Lab Components of EC (9/9) Initialization usually done at random  Need to ensure even spread and mixture of possible allele values  Can include existing solutions, or use problem-specific heuristics, to “ seed ” the population Termination condition checked every generation  Reaching some (known/hoped for) fitness  Reaching some maximum allowed number of generations  Reaching some minimum level of diversity  Reaching some specified number of generations without fitness improvement 19

(C) SNU CSE Biointelligence Lab Paradigms in EC Evolutionary Programming (EP)  [L. Fogel et al., 1966]  FSMs, mutation only, tournament selection Evolution Strategy (ES)  [I. Rechenberg, 1973]  Real values, mainly mutation, ranking selection Genetic Algorithm (GA)  [J. Holland, 1975]  Bitstrings, mainly crossover, proportionate selection Genetic Programming (GP)  [J. Koza, 1992]  Trees, mainly crossover, proportionate selection 20

Appendix: Evolution Strategy

Evolution Strategy (ES) Problem of real-valued optimization Find extremum (minimum) of function F(X): R n ->R Operate directly on real-valued vector X Generate new solutions through Gaussian mutation of all components Selection mechanism for determining new parents (C) SNU CSE Biointelligence Lab22

ES: Representation One individual: The three parts of an individual: : Object variables : Standard deviations : Rotation angles Fitness Variances Covariances (C) SNU CSE Biointelligence Lab23

, where r x, r , r   {-, d, D, i, I, g, G}, e.g. r dII ES: Operator - Recombination (C) SNU CSE Biointelligence Lab24

m { ,  ’,  } : I  I is an asexual operator.  n  = n, n  = n(n-1)/2  1 < n  < n, n  = 0  n  = 1, n  = 0 ES: Operator - Mutation (C) SNU CSE Biointelligence Lab25

ES: Illustration of Mutation Hyperellipsoids Line of equal probability density to place an offspring (C) SNU CSE Biointelligence Lab26

ES: Evolution Strategy vs. Genetic Algorithm Create random initial population Evaluate population Select individuals for variation Vary Insert into population Create random initial population Evaluate population Vary Selection Insert into population (C) SNU CSE Biointelligence Lab27

Part II: Genetic Programming

Overview Genetic Programming Tree-based Representations Setting Up a Genetic Programming Run Example: Wall-Following Robot Result 29(C) SNU CSE Biointelligence Lab

Genetic Programming (GP) GP is an domain- independent method that evolves a population of programs to solve a problem. GP uses variable-size tree- representations rather than fixed-length strings of binary values in typical GA. GP has been successful in many domains. 30

(C) SNU CSE Biointelligence Lab Tree-based Representations (1/6) Function set: internal nodes  Functions, predicates, or actions which take one or more arguments Terminal set: leaf nodes  Program constants, actions, or functions which take no arguments S-expression: (+ 3 (/ (  5 4) 7)) Terminals = {3, 4, 5, 7} Functions = {+, , /} 31

(C) SNU CSE Biointelligence Lab Tree-based Representations (2/6) Trees can represent:  Arithmetic formula  Logical formula  Program  And many others depending on the definition of terminals and nonterminals. (x  true)  (( x  y )  (z  (x  y))) i =1; while (i < 20) { i = i +1 } 32

(C) SNU CSE Biointelligence Lab Tree-based Representations (3/6) 33

(C) SNU CSE Biointelligence Lab Tree-based Representations (4/6) (x  true)  (( x  y )  (z  (x  y))) 34

(C) SNU CSE Biointelligence Lab Tree-based Representations (5/6) i =1; while (i < 20) { i = i +1 } 35

(C) SNU CSE Biointelligence Lab Tree-based Representations (6/6) In GA, ES, EP chromosomes are linear structures (bit strings, integer string, real-valued vectors, permutations) Tree shaped chromosomes are non-linear structures. In GA, ES, EP the size of the chromosomes is fixed. Trees in GP may vary in depth and width. 36

Setting up a GP Run Users need to specify: 1.The terminal / function set 2.The fitness measure 3.Algorithm parameters to control the run  Population size, maximum number of generations  Selection / variation operators and their probabilities  The termination criterion  Maximum depth of a GP tree, etc. 37(C) SNU CSE Biointelligence Lab

Example: Wall-Following Robot (Step 1) Program Representation in GP  Functions  AND (x, y) = 0 if x = 0; else y  OR (x, y) = 1 if x = 1; else y  NOT (x) = 0 if x = 1; else 1  IF (x, y, z) = y if x = 1; else z  Terminals  Actions: move the robot one cell to each direction { north, east, south, west }  Sensory input: its value is 0 whenever the coressponding cell is free for the robot to occupy; otherwise, 1. { n, ne, e, se, s, sw, w, nw } 38

(C) SNU CSE Biointelligence Lab Program Example 39

Example: Wall-Following Robot (Step 1) Be careful when specifying function / terminal set. To avoid syntactic error  All programs in the initial population should be valid, executable programs.  The genetic operators during the run should produce valid, executable programs as offspring. To avoid run-time error  All functions can take any terminal or the results produced by any other functions as input. 40(C) SNU CSE Biointelligence Lab

Example: Wall-Following Robot (Step 2) Fitness measures can be  Error between program’s output and the desired output.  Accuracy in recognizing patterns or classifying objects into classes.  Payoff a game-playing program produces. Often each program is executed over a representative sample of fitness cases. In our wall-following robot problem  the number of cells next to the wall that are visited during 60 steps over 10 independent runs.  Perfect score (320) : One Run (32)  10 randomly chosen starting points 41(C) SNU CSE Biointelligence Lab

Example: Wall-Following Robot (Step 3) Genetic operators – crossover (subtree exchange) + b   ab + b +   aab +  ab   ab + b +  a b  42(C) SNU CSE Biointelligence Lab

Example: Wall-Following Robot (Step 3) Genetic operators – mutation (C) SNU CSE Biointelligence Lab43  ab + b /  a +  b + b /  a - ba

Example: Wall-Following Robot (Step 3) Population size: 5,000 Termination condition: found perfect solution Creating Next Generation  500 programs (10%) are copied directly into next generation (elitism).  Tournament selection 7 programs are randomly selected from the population 5,000. The most fit of these 7 programs is chosen.  4,500 programs (90%) are generated by crossover.  Parents are each chosen by tournament selection.  In this example, mutation was not used. (C) SNU CSE Biointelligence Lab44

Example: Wall-Following Robot (Step 3) (C) SNU CSE Biointelligence Lab45

(C) SNU CSE Biointelligence Lab Result (1/5) Generation 0  The most fit program (fitness = 92)  Starting in any cell, this program moves east until it reaches a cell next to the wall; then it moves north until it can move east again or it moves west and gets trapped in the upper-left cell. 46

(C) SNU CSE Biointelligence Lab Result (2/5) Generation 2  The most fit program (fitness = 117)  Smaller than the best one of generation 0, but it does get stuck in the lower-right corner. 47

(C) SNU CSE Biointelligence Lab Result (3/5) Generation 6  The most fit program (fitness = 163)  Following the wall perfectly but still gets stuck in the bottom- right corner. 48

(C) SNU CSE Biointelligence Lab Result (4/5) Generation 10  The most fit program (fitness = 320)  Following the wall around clockwise and moves south to the wall if it doesn’t start next to it. 49

(C) SNU CSE Biointelligence Lab Result (5/5) Fitness Curve  Fitness as a function of generation number  The progressive (but often small) improvement from generation to generation 50

Part III: Advanced Topics and Applications

Overview Theoretical topics  Mathematical Models of Simple Genetic Algorithms  Bloat Phenomena in Genetic Programming Applications Summary 52(C) SNU CSE Biointelligence Lab

Mathematical Models of Simple GAs (1/7) Based on (Vose & Liepins, 1991). The model simple GA 1.Start with a random population of binary strings of length l. 2.Calculate the fitness f(x) of each string x in the population. 3.Choose (with replacement) two parents from the population with probability proportional to each string’s relative fitness in the population. 4.Crossover the two parents (at single random point) with probability p c to from two offspring and discard one at random. 5.Mutate each bit in the selected offspring with probability p m and replace it in the new population. 6.Go to Step 3 until new population is complete. 7.Go to Step 1. 53(C) SNU CSE Biointelligence Lab

Notations  Each string is represented by binary string between 0 and 2 l -1. Mathematical Models of Simple GAs (2/7) Proportion of string i in the population The selection probability of string i, assuming that fitness is equal to the number of 1’s in the string. 54(C) SNU CSE Biointelligence Lab

Mathematical Models of Simple GAs (3/7) GA with selection only.  Expectation of next population is equal to the selection probabilities. 55(C) SNU CSE Biointelligence Lab

Mathematical Models of Simple GAs (4/7) Including crossover and mutation.  Define recombination matrix M for crossover and mutation.  r i,j (k) – the probability that string k will be produced, given that string i and j are selected to mate.  Defining r i,j (k) and M is tricky. We will show defining r i,j (0) only. 56(C) SNU CSE Biointelligence Lab

Mathematical Models of Simple GAs (5/7) Defining r i,j (0)  Case 1: crossover does not occur and the selected offspring is mutated to all zeros.  Case 2: crossover occurs and the selected offspring is mutated to all zeros. Case 1  |i|: the number of 1’s in a string i.  The probability that i be mutated to all zeros  57(C) SNU CSE Biointelligence Lab

Mathematical Models of Simple GAs (6/7) Case 2  Temporary definition  |h|=|i|-|i 2 |+|j 2 | & |k|=|j|-|j 2 |+|i 2 |.  But, gives the number of 1’s in rightmost c bits of i.  Thus, i: j: h: k: i1i1 i2i2 j1j1 j2j2 i: : (C) SNU CSE Biointelligence Lab

Mathematical Models of Simple GAs (7/7) For the rest of derivation of r i,j (0) and M, refer to (Vose & Liepins, 1991). 59(C) SNU CSE Biointelligence Lab

Bloat Phenomena in GP (1/3) Bloat  Program growth without (significant) return in terms of fitness. Fitness Program size 60(C) SNU CSE Biointelligence Lab

Bloat Phenomena in GP (2/3) Why does it happen?  No certain explanation.  Replication accuracy theory  GP evolves towards (bloated) representations that increase replication accuracy.  Crossover bias theory  Short programs are less fit than long ones  long programs have a selective advantage  bloat 61(C) SNU CSE Biointelligence Lab

Bloat Phenomena in GP (3/3) Techniques limiting bloat  Fixed size or depth limit: programs exceeding the limit are discarded and a parent is kept.  Dangerous: acts as advantage for programs that violate the limit.  Parsimony pressure: a penalty term for larger programs is added to the fitness value.  (Fitness_parsimony) = (Fitness_raw) – c * (program_size) 62(C) SNU CSE Biointelligence Lab

Part IV: Applications

(C) SNU CSE Biointelligence Lab Applications of EC Numerical, Combinatorial Optimization System Modeling and Identification Planning and Control Engineering Design Data Mining Machine Learning Artificial Life 64

Co-evolving Soccer Softbots (1/5) At RoboCup there are two "leagues": the "real" robot league and the "virtual" softbot league How do you do this with GP?  GP breeding strategies: homogeneous and heterogeneous  Decision of the basic set of function with which to evolve players  Creation of an evaluation environment for our GP individuals Application Example 1 Co-evolving Soccer Softbots With Genetic Programming (C) SNU CSE Biointelligence Lab65

Co-evolving Soccer Softbots (2/5) Initial Random Population Application Example 1 (C) SNU CSE Biointelligence Lab66

Co-evolving Soccer Softbots (3/5) Kiddie Soccer Application Example 1 (C) SNU CSE Biointelligence Lab67

Co-evolving Soccer Softbots (4/5) Learning to Block the Goal Application Example 1 (C) SNU CSE Biointelligence Lab68

Co-evolving Soccer Softbots (5/5) Becoming Territorial Application Example 1 (C) SNU CSE Biointelligence Lab69

Evolutionary Arts Exploits the process of evolution to create an artwork according to an evolutionary algorithm. Selection may be done by the viewer explicitly by selecting individuals which are aesthetically pleasing, or implicitly according to the length of time a viewer spends near a piece of evolving art. EvoMUSART workshops 2009  EvoMUSART/EvoMUSART.html Application Example 2 70(C) SNU CSE Biointelligence Lab

Part V: Summary & Further Information

(C) SNU CSE Biointelligence Lab Recapitulation of EA EAs fall into the category of “ generate and test ” algorithms. They are stochastic, population-based algorithms. Variation operators (recombination and mutation) create the necessary diversity and thereby facilitate novelty. Selection reduces diversity and acts as a force pushing quality. Fundamental trade-off between exploration and exploitation 72

Exploration vs. Exploitation There are many possibilities to set the parameters to increase the explorative capability of an EA  Increase variation probability  Increase mutation rate  Increase crossover rate  Increase population size  Decrease selection pressure  Add noise to fitness function  Add noise in selection  Increase rank ratio in ranking selection  Decrease tournament size in tournament selection Important is the combined effect of various parameter settings. (C) SNU CSE Biointelligence Lab73

(C) SNU CSE Biointelligence Lab Typical Behavior of an EA Phases in optimizing on a 1-dimensional fitness landscape Early phase: quasi-random population distribution Mid-phase: population arranged around/on hills Late phase: population concentrated on high hills 74

Are Long Runs Beneficial? (C) SNU CSE Biointelligence Lab75 Best fitness in population Time (number of generations) Progress in 1 st half Progress in 2 nd half Answer: - It depends how much you want the last bit of progress - It may be better to do more shorter runs

(C) SNU CSE Biointelligence Lab Scale of “all” problems Performance of methods on problems Random search Special, problem tailored method Evolutionary algorithm EAs as Problem Solvers 76

(C) SNU CSE Biointelligence Lab Advantages of EC No presumptions w.r.t. problem space Widely applicable Low development & application costs Easy to incorporate other methods Solutions are interpretable (unlike NN) Can be run interactively, accommodate user proposed solutions Provide many alternative solutions 77

(C) SNU CSE Biointelligence Lab Disadvantages of EC No guarantee for optimal solution within finite time Weak theoretical basis May need parameter tuning Often computationally expensive, i.e. slow 78

(C) SNU CSE Biointelligence Lab Further Information on EC Conferences  IEEE Congress on Evolutionary Computation (CEC)  Genetic and Evolutionary Computation Conference (GECCO)  Parallel Problem Solving from Nature (PPSN)  Foundation of Genetic Algorithms (FOGA)  EuroGP, EvoCOP, and EvoWorkshops  Int. Conf. on Simulated Evolution and Learning (SEAL) Journals  IEEE Transactions on Evolutionary Computation  Evolutionary Computation  Genetic Programming and Evolvable Machines 79

(C) SNU CSE Biointelligence Lab References Evolutionary Computation by Kenneth A. De Jong, Evolutionary Computation Evolutionary Computation: Toward a New Philosophy of Machine Intelligence, Third Edition (IEEE Press Series on Computational Intelligence) by David B. Fogel, 2005, Evolutionary Computation: Toward a New Philosophy of Machine Intelligence, Third Edition (IEEE Press Series on Computational Intelligence) Introduction to Evolutionary Computing by A. E. Eiben and J. E Smith, Springer, Genetic Programming: On the Programming of Computers by Means of Natural Selection (Complex Adaptive Systems) by John R. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection (Complex Adaptive Systems) Genetic Programming: An Introduction by W. Banzhaf, P. Nordin, R. Keller, and F. Francone, Morgan Kaufmann, A Field Guide to Genetic Programming by R. Poli and W. B. Langdon, Online free book – 80