Dynamic Optimisation So far have looked at the use of EA’s as function optimisers for stationary functions, i.e. where the fitness landscape remains static w.r.t. time. Many interesting applications are non- stationary: given a potential solution x and a fitness function f, then f(x, t1) and f(x,t2) may differ.

Types of non-stationary function Cobb (1990) defines two ways of classifying non-stationary functions –Switching vs. Continuous based on time-scale of change w.r.t. rate of evaluation –Markovian vs. State Dependant f(x,t) comes from asymptotic distribution or is strictly a function of time.

Strategies Used Cobb(1990) identifies two different classes of strategies for NS environments 1: Expand the memory of the EA. e.g. Goldberg & Smith (87), Smith (88), Dasgupta & MacGregor(1992). good solution if there are a fixed number of states in MSE 2: (Adaptively) expand variation in the population

Experiments with Std GA Cobb (1990) observed the behaviour of a standard GA on a parabolic function with a the optima moving sinusoidally in space. Offline performance decreases as rate of change increases for all p_m As rate of change increases the value of p_m which gives optimal offline performance increases As problem difficulty increased, rate of change that GA could track decreased

Triggered Hyper Mutation(HM) Cobb proposed adaptive mutation mechanism with two mutation rates 0.5/0.001 Higher rate triggered by drop in fitness of best in generation Tracked optima far better than SGA, especially when the time-optima were spatially close

Pitfalls with HM Grefenstette (1992) noted that under certain conditions HM might never get triggered. Function composed of 14 sinusoidal hills initial optima 5.0, subsequent optima with height 10.0, appears after 20 generations, and moves every 20 generations SGA goes to 5.0, rises to 10.0 when optima close but does not track HM, goes to 5, only tracks 10.0 after optima moves close to original peak, then triggering works

Random Immigrants (RI) RI strategy proposed, replace proportion of population with randomly created members at every generation –30% replacement gives best off-line tracking –too high and it can’t converge between changes –however off-line performance decreases with proportion replaced

Comparison of techniques Cobb & Grefenstette (1993) compared HM vs. RI vs SGA (with high mutation rate) Noted difference in nature of mutation –SGA uniform in pop and time –HM uniform in pop, not in time –RI uniform in time, not in pop. NB this (later) version of HM triggered by decrease in running mean over 5 generations of best in pop.

Comparison Experiments 2 landscapes, A as per picture, B composed of various shaped hills. 32 bits each 3 types of change: –linear motion in A, moving 1 step along an axis every 2/5 generations –randomly shifting the optima in A every 20 generations –swap between A and B every 2/20 generations

Comparison Conclusions ran lots of experiments to find good settings for each algorithm with each problem SGA: – p_m 0.1 was reasonably good at the translation tasks, but gave v. poor online performance. –couldn’t track moving optimum or oscillation – p_m needs to be matched to degree of change

Comparison Conclusions cont.. HM –High variance in performance noted –HM rate needed tuning a lot to problem –Much better at tracking sudden changes than SGA, and gave better online performance than SGA or RI`when rate of change was slow enough to allow lower rate of mutation

Comparison Conclusions cont. RI –not very good at tracking linear movement –best at the oscillating task –they hypothesise ( but don’t show) that this is because it allows the preservation of niches –poor performance on stationary problems and slow movements

Vavak 1993-> Concentrated on on-line performance, for control of industrial systems Compared generational vs. steady state GA’s on a 1 dimensional moving problem –SSGA better from random or converged population. –entropy reacts faster and reaches lower value for SSGA

Design choices for SSEAs Smith & Vavak showed choice of replacement strategy was critical Best algorithms –Use elitism, but –Systematically re-evaluate all points in population –Overall a “conservative” delete-oldest policy was found to work best,

Self-Adaptive Approaches Back et al showed that it is not necessary to use a trigger if you have a self-adaptive system. ES was able to track a moving optima reliably

Co-evolution Sometimes the fitness of a solution may not be externally defined, e.g.: –Individuals compete against others from same population E.g. evolving game playing strategies –Individuals must be evaluated in context of solutions from other populations representing Other parts of decomposed problem Competing species Obviously in biology the fitness in our “adaptive landscape” metaphor is a function of the other competing species rather than being fixed.

Some terminology Mutualism or Symbiosis: –The beneficial co-adaptation of different species, may become co- located (e.g. gut flora) –In EC known as Co-operative Coevolution: e.g. lots of work by Bull, Potter and DeJong on function decomposition Predation or Parasitism: –Antagonistic co-adaption e.g. fox/rabbit –Also known as competitive coevolution: e.g. lots of work on test- soluton problems, game-playing etc. Note that both types of coevolution can occur in single population or multi-population forms

How to evaluate fitness? Single population: –Test each solution against every other (or a randomly selected sample if pop. is large) Multiple populations –Need to decide a pairing strategy e.g.: Evolve populations in turn, use best of other pops. Pair with randomly chosen members of other pops –Lots of work by Bull, Husbands verdict still out –Neat solution by Husbands uses cellular model with one of each pop on each grid point –Paredis uses “Life-Time-Fitness-Evalaution” – mean of last twenty encounters with another species

Some examples Hillis 1993: –Co-evolved sorting networks with “difficult” test sequences - found network of smallest size known at the time Axelrod: –Organised competitions for Iterated Prisoners Dilemma Showed he could evolve winning strategies Paredis: –Coevolved solutions and their representations to remove operator bias –Also used coevolution for constraint satisfaction problems

This sounds great, why don’t we all do it …. Desirable behaviour in competitive evolution is known as “arms races” –Red queen effect: run faster and faster just to keep up But problems arise: –Cycling –Mediocre stable states –disengagement

What does the future hold Several groups actively researching solutions to these problems LOTS of potential applications: –Agent based systems for e-commerce, trading, game-playing etc. –Investigating difficult problems in abstract fields eg maths –Evolving team behaviour –Optimisation with (automatic? ) problem decomposition