Evolutionary Game Theory Amit Bahl CIS620

Outline zEGT versus CGT zEvolutionary Stable Strategies Concepts and Examples zReplicator Dynamics Concepts and Examples zOverview of 2 papers Selection methods, finite populations

EGT v. Conventional Game Theory zModels used to study interactive decision making. zEquilibrium is still at heart of the model. zKey difference is in the notion of rationality of agents.

Agent Rationality zIn GT, one assumes that agents are perfectly rational. zIn EGT, trial and error process gives strategies that can be selected for by some force (evolution - biological, cultural, etc…). zThis lack of rationality is the point of departure between EGT and GT.

Evolution zWhen in biological sense, natural selection is mode of evolution. zStrategies that increase Darwinian fitness are preferable. zFrequency dependent selection.

Evolutionary Game Theory (EGT) Has origins in work of R.A. Fisher [The Genetic Theory of Natural Selection (1930)]. Fisher studied why sex ratio is approximately equal in many species. Maynard Smith and Price introduce concept of an ESS [The Logic of Animal Conflict (1973)]. Taylor, Zeeman, Jonker ( ) provide continuous dynamics for EGT (replicator dynamics).

ESS Approach zESS = Nash Equilibrium + Stability Condition zNotion of stability applies only to isolated bursts of mutations. zSelection will tend to lead to an ESS, once at an ESS selection keeps us there.

ESS - Definition Consider a 2 player symmetric game with ESS given by I with payoff matrix E. Let p be a small percentage of population playing mutant strategy J  I. Fitness given by W(I) = W 0 + (1-p)E(I,I) + pE(I,J) W(J) = W 0 + (1-p)E(J,I) + pE(J,J) Require that W(I) > W(J)

ESS - Definition zStandard Definition for ESS (Maynard Smith). zI is an ESS if for all J  I, E(I,I)  E(J,I) and E(I,I) = E(J,I)  E(I,J) > E(J,J) where E is the payoff function.

ESS - Definition Assumptions: 1) Pairwise, symmetric contests 2) Asexual inheritance 3) Infinite population 4) Complete mixing

ESS - Existence zLet G be a two-payer symmetric game with 2 pure strategies such that E(s1,s1)  E(s2,s1) AND E(s1,s2)  E(s2,s2) then G has an ESS.

ESS Existence zIf a > c, then s1 is ESS. zIf d > b, then s2 is ESS. zOtherwise, ESS given by playing s1 with probability equal to (b-d)/[(b-d)+(a-c)].

ESS - Example 1 zConsider the Hawk-Dove game with payoff matrix zNash equilibrium given by (7/12,5/12). zThis is also an ESS.

ESS - Example 1 zBishop-Cannings Theorem: If I is a mixed ESS with support a,b,c,…, then E(a,I) = E(b,I) = … = E(I,I). zAt a stable polymorphic state, the fitness of Hawks and Doves must be the same. zW(H) = W(D) => The ESS given is a stable polymorphism.

Stable Polymorphic State

ESS - Example 2 zConsider the Rock-Scissors-Paper Game. zPayoff matrix is given by R S P R -e 1 -1 S -1 -e 1 P e zThen I = (1/3,1/3,1/3) is an ESS but stable polymorphic population 1/3R,1/3P,1/3S is not stable.

ESS - Example 3 zPayoff matrix : zThen I = (1/3,1/3,1/3) is the unique NE, but not an ESS since E(I,s1)=E(s1,s1)= 1.

Sex Ratios zRecall Fisher’s analysis of the sex ratio. zWhy are there approximately equal numbers of males and females in a population? zGreatest production of offspring would be achieved if there were many times more females than males.

Sex Ratios zLet sex ratio be s males and (1-s) females. zW(s,s’) = fitness of playing s in population of s’ zFitness is the number of grandchildren zW(s,s’) = N 2 [(1-s) + s(1-s’)/s’] W(s’,s’) = 2N 2 (1-s’) zNeed s* s.t.  s W(s*,s*)  W(s,s*)

Dynamics Approach zAims to study actual evolutionary process. zOne Approach is Replicator Dynamics. zReplicator dynamics are a set of deterministic difference or differential equations.

RD - Example 1 zAssumptions: Discrete time model, non- overlapping generations. zx i (t) = proportion playing i at time t z  (i,x(t)) = E(number of replacement for agent playing i at time t) z  j {x j (t)  (j,x(t))} = v(x(t)) zx i (t+1) = [x i (t)  (i,x(t))]/ v(x(t))

RD - Example 1 zAssumptions: Discrete time model, non- overlapping generations. zx i (t+1) - x i (t) = x i (t) [  (i,x(t)) - v(x(t))] v(x(t))

RD - Example 2 zAssumptions : overlapping generations, discrete time model. zIn time period of length , let fraction  give birth to agents also playing same strategy. z  j x j (t)[1 +   (j,x(t))] = v(x(t)) zx i (t+  ) = x i (t)[1 +   (i,x(t))] v(x(t))

RD - Example 2 zAssumptions : overlapping generations, discrete time model. zx i (t+  ) - x i (t)= x i (t)[   (i,x(t)) -  v(x(t))] 1+  v(x(t))

RD - Example 3 zAssumptions: Continuous time model, overlapping generations. zLet   0, then dx i /dt = x i (t)[  (i,x(t)) - v(x(t))]

Stability zLet x(x 0,t):  S X R  S be the unique solution to the replicator dynamic. zA state x   S is stationary if dx/dt = 0. zA state x is stable if it is stationary and for every neighborhood V of x, there exists a U  V s.t.  x 0  U,  t x(x 0,t)  V.

Propostions for RD zIf (x,x) is a NE, then x is a stationary state of the RD. ydx i /dt = x i (t)[  (i,x(t)) - v(x(t))] zWhat about the converse? yConsider population of all doves.

Propostions for RD zIf x is a stable state of the RD, then (x,x) is a NE. yConsider any perturbation that introduces a better reply. zWhat about the converse? Consider:

Stronger notion of Stability zA state x is asymptotically stable if it is stable and there exists a neighborhood V of x s.t.  x 0  V, lim t  x(x 0,t) = x.

ESS and RD zIn general, every ESS is asymptotically stable. zWhat about the converse?

ESS and RD zConsider the following game: zUnique NE given by x* = (1/3,1/3,1/3). zIf x = (0,1/2,1/2), then E(x,x*)=E(x*,x*)=2/3 but E(x,x)=5/4 > 7/6=E(x*,x).

ESS and RD In 2X2 games, x is an ESS if and only if x is asymptotically stable.

A Game-Theoretic Investigation of Selection Methods Used in Evolutionary Algorithms Ficici, Melnik, Pollack

Selection Methods zHow do common selection methods used in evolutionary algorithms function in EGT? zDynamics and fixed points of the game.

Selection function zx i (t+1) = S(F(x i (t)),x i (t)) where S is the selection function, F is the fitness function, and x i (t) is the proportion of population playing i at time t.

Fitness Dependent Selection f’ = (p X f)/(p f) {x(x 0,t): t  R} = orbit passing through x 0.

Truncation Selection 1) Sort by fitness 2) Replace k% of lowest by k% of highest

Truncation Selection zConsider the Hawk-Dove game with ESS given by (7/12 H, 5/12 D) If.5 < x H (t) < 7/12, then x H (t+1) = 1.

Truncation Selection Map Diagram:

( , )-ES Selection zGiven a population of offspring, the best  are chosen to parent the next generation. zMore extreme than truncation selection.

Linear Rank Selection zAgents sorted according to fitness. zAssigned new fitness values according to their rank. zCauses fitness to change linearly with rank. zCauses cycles around ESS.

Linear Rank Selection Map Diagram:

Boltzman Selection zInspired by simulated annealing. zSelection pressure slowly increased over time to focus search. zIn some cases, BS is able to retain the dynamics and equilibria in EGT.

Boltzman Selection Map Diagram:

Effects of Finite Populations on Evolutionary Stable Strategies. Ficici, Pollack

Finite Populations zEffects of finite population on EGT. zBegin at ESS (7/12,5/12) and test n=60,120,300,600, and 900 for 2000 generations. z100 replicates of each experiment.

Finite Populations zResults:

Convergence zFor a n player name, consider the MC with states given by #of hawks. zDefine transition matrix P. zb t = b 0 P t zE(x H (t)) = (1/n)  n i=0 b H t * i zlim t  E(x H (t)) = b  H zEstimate | E(x H (t)) - b  H |

Convergence Simulation zResults: