1. Presenter: Chih-Yuan Chou GA-BASED ALGORITHMS FOR FINDING EQUILIBRIUM 1

2. AGENDA “The evolution of strategies in the iterated prisoner's dilemma” (Axelrod, 1987). “Genetic algorithm learning and evolutionary games” (Riechmann, 2001). “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004). 2

3. AGENDA “The evolution of strategies in the iterated prisoner's dilemma” (Axelrod, 1987). “Genetic algorithm learning and evolutionary games” (Riechmann, 2001). “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004). 3

4. PRISONER'S DILEMMA 4

5. ITERATED PRISONER'S DILEMMA The same two individuals may meet more than once. An individual can recognize a previous interactant and remember some aspects of the prior outcomes. A strategy is a decision rule which specifies the probability of cooperation or defection as a function of the history of the interaction so far. 5

6. 5-STAGE SIMULATION 1. Initialize the population by generate random strings of C's and D's. 2. each individual player uses the strategy defined by its chromosome to play in iterated Prisoner's Dilemma with other strategies 3. The relatively successful individuals are selected to have more offspring. 6

7. 5-STAGE SIMULATION 4. crossover and mutation 5. The new population will display patterns of behavior that are more like those of the successful individuals of the previous generation. 7

8. SIMULATION RESULTS Cooperate after three mutual cooperations. Defect after three mutual defections. Defect when the other player defects out of the blue. Continue to cooperate after cooperation has been restored. 8

9. TIT FOR TAT Cooperate on the first move Do whatever the other player did on the preceding move 9

10. AGENDA “The evolution of strategies in the iterated prisoner's dilemma” (Axelrod, 1987). “Genetic algorithm learning and evolutionary games” (Riechmann, 2001). “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004). 10

11. CANONICAL GA 11

12. ECONOMIC GA Definition: An economic genetic algorithm is a genetic algorithm with a state-dependent fitness function. The fitness of an economic agent does not only depend on her own strategy, but also on the strategies of all other agents involved in the model. 12

13. GENETIC POPULATIONS AS NEAR NASH EQUILIBRIUM A population can be described by counting how often each of the different possible strategies is used by the members of the population. The population tends over time to move to a Nash equilibrium without fully reaching it. 13

14. EVOLUTIONARILY STABLE STATES (ESS) a strategy is evolutionarily stable if, relative to its population, it performs better than any ‘invading’ strategy. ESS are based on the notion that invading ‘mutant’ strategies are somehow rejected or eliminated from the population. 14

15. SET UP A CONCEPT OF EVOLUTIONARY STABILITY FOR GA A genetic population is evolutionarily stable if the process of the genetic algorithm rejects an invasion by one or more strategies from the genetic population. Invasion itself can either take the form of a totally new strategy entering the population or it can simply mean a change in the frequency of the strategies already contained within the population. A more precise definition of an evolutionarily stable population might be: A population is evolutionarily stable if it is resistant against changes in its composition. 15

16. SET UP A CONCEPT OF EVOLUTIONARY STABILITY FOR GA If a genetic population is evolutionarily stable, it will recover from an invasion within a finite number of steps of the GA, which means that in the long run the population will not lastingly be changed. 16

17. CONCLUSION OF THIS PAPER Economic genetic algorithm learning can be shown to be a specific form of an evolutionary game. The results of this paper demonstrate that research in the dynamics of economic GA learning models can be equipped with the whole tool box of evolutionary game theory. 17

18. AGENDA “The evolution of strategies in the iterated prisoner's dilemma” (Axelrod, 1987). “Genetic algorithm learning and evolutionary games” (Riechmann, 2001). “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” (Son, 2004). 18

19. MOTIVATION The conventional NE search algorithms and coevolutionary algorithms can not differentiate real NE and “local NE traps” 19

20. CONTRIBUTIONS OF THIS PAPER Prove that any conventional iterative NE search algorithms applied to games with local optima can misidentify NE by following a local optimization path. Introduce the concept of “local NE trap” that attracts the conventional iterative NE search algorithms based on local optimization tools. Develop an enhanced version of coevolutionary programming to overcome the problem. 20

21. LOCAL NE TRAP 21

22. LOCAL NE TRAP 22

23. LOCAL NE TRAP 23

24. LOCAL NE TRAP 24

25. MODEL OF ITERATIVE NE SEARCH ALGORITHMS 1) Initialize each player’s strategy choice set. 2) Given other players’ decisions, solve each player’s profit maximization problem. Do this step for all players. 3) Repeat Step 2) until the NE condition is satisfied or the maximum iteration number is reached. 25

26. MODEL OF ITERATIVE NE SEARCH ALGORITHMS Proposition: Every local NE, including any local NE trap, can be the solution of any conventional iterative NE search algorithm. 26

27. EXAMPLE 27

28. EXAMPLE 28

29. COEVOLUTIONARY PROGRAMMING Each generation of the genetic algorithm-based coevolutionary programming consists of two parts: The random matching The evolutionary process of each player 29

30. COEVOLUTIONARY PROGRAMMING 30

31. COEVOLUTIONARY PROGRAMMING 31

32. WHOLE PROCESS OF COEVOLUTIONARY PROGRAMMING 32 1) Set the basic parameters of maximum generation number, population size, crossover rate, and mutation rate. 2) Initialize the strategy population of each player. 3) Choose one strategy from the strategy population of each player randomly from among the strategies that have not already been assigned profits. Input the strategy information to the tournament. The result of the tournament will decide profit values for these chosen strategies. 4) Repeat Step 3) until every strategy is assigned a profit value. 5) Apply the evolutionary operators to each strategy population. Keep the best strategy of the current generation alive (elitism). 6) Repeat Steps 3)–5) until the maximum generation number is reached.

33. HYBRID COEVOLUTIONARY PROGRAMMING “best rival matching and fine tuning.” The chosen individual will be matched against the best strategy of the other populations in the current generation. 33

34. THE ALGORITHMIC STEPS 1)Randomly choose a strategy from the strategy population of a player. 2)Take the best strategies of all other rivals from the previous generation. 3)Apply a local hill climber for fine tuning of the chosen strategy of the player. 4)Repeat Steps 1)–3) as many times as set by the best rival matching rate. For example, with a best rival matching rate of 0.2, 20% of strategies in each population will be chosen. 5) Repeat Steps 1)–4) for every other player in turn. 34

35. THE ALGORITHMIC STEPS 35

36. SIMULATION This paper compared three algorithms: Conventional NE search algorithm Simple coevolutionary programming Hybrid coevolutionary programming 36

37. GAME CONFIGURATION The profit functions for player A and player B: 37

38. GAME CONFIGURATION NE and local NE trap of the numerical game example 38

39. CONVENTIONAL NE SEARCH Find multiple solutions including NE and local NE trap without being able to differentiate the NE from the local NE traps. 39

40. LOCAL ITERATIVE SEARCH Iterative NE search result 40

41. COEVOLUTIONARY PROGRAMMING 41

42. HYBRID COEVOLUTIONARY PROGRAMMING 42

43. CONCLUSION OF THIS PAPER 43

44. CONCLUSION OF THIS PAPER Prove that any iterative NE search algorithms based on local optimization can not differentiate between the real NE and the “local NE traps” Suggested an enhancement of coevolutionary programming to solve games with local optima. Hybrid coevolutionary programming converged to the NE, if it existed, and did not converge to any local NE traps. 44

45. REFERENCE “The evolution of strategies in the iterated prisoner's dilemma” Axelrod, 1987. “Genetic algorithm learning and evolutionary games” Riechmann, 2001. “Hybrid coevolutionary programming for Nash equilibrium search in games with local optima” Son, 2004. 45