林偉楷 Taiwan Evolutionary Intelligence Laboratory

Outline  Introduction  Mixed strategy games, less co-evolvable  Pure strategy games, more co-evolvable Needle-in-a-haystack game Simple state game Coevolutionary GA with niching The optimal number of opponents  A less co-evolvable pure strategy game  Conclusions The Co-Evolvability of Games in Coevolutionary Genetic Algorithms2

Introduction: Coevolution (1)  Coevolutionary algorithm evaluates the fitness of a solution by other candidate solutions. Coevolutionary fitness is absolute or subjective Traditional EC fitness is relative or objective  No objective measures exist  Objective measure difficult to formalize or unknown The Co-Evolvability of Games in Coevolutionary Genetic Algorithms3

Introduction: Coevolution (2) Ref: de Jong et al. Introductory tutorial on coevolution. GECCO ’07. The Co-Evolvability of Games in Coevolutionary Genetic Algorithms4 CEA Population Player 1 Player 2 Game Playing Strategies

Introduction: Coevolution (3)  Coevolution can also be applied to certain types of structure in search space The Co-Evolvability of Games in Coevolutionary Genetic Algorithms5 CEA Population Agent 1 Population Agent 2 Population Agent 3 Multiagent Teams

Introduction: Pure Strategies  In game theory, a game can be defined by a set of players, the valid strategies of each player, and the payoff of all players in each strategy profile.  Pure strategy refers to such deterministic strategy (and thus payoff).  Example: rock, scissor and paper are 3 pure strategies. The Co-Evolvability of Games in Coevolutionary Genetic Algorithms6

Introduction: Mixed Strategies  In contrast to pure strategy, a mixed strategy plays each pure strategy with a probability.  The payoff is than considered as an expected value.  Example: randomly plays rock, scissor and paper with probability 0.4, 0.3 and 0.3. The Co-Evolvability of Games in Coevolutionary Genetic Algorithms7

Introduction: Some Properties of Games We Interested In  Some properties are common in real- world games: Two-player: games involving more players are very complex Zero-sum: no win-win strategy Symmetric: both player are unbiased The Co-Evolvability of Games in Coevolutionary Genetic Algorithms8

Mixed Strategy, Less Co-evolvable  For a two-player, zero-sum and symmetric game with a non-pure mixed strategy Nash equilibrium, the equilibrium strategy does not get a higher payoff than other strategies. The Co-Evolvability of Games in Coevolutionary Genetic Algorithms9

Rock-paper-scissors  In a population with equal probability to play R/P/S in average, different strategies have the payoff: The Co-Evolvability of Games in Coevolutionary Genetic Algorithms10 StrategyThe probability of each outcome RockPaperScissorWinTieLose 1/ All their linear combinations1/3

Turn-Based Games  We focus on turn-based games, which has huge solution space.  Turn-based games can be modeled in a state-transition concept: for each state, the player makes an action and it goes to another state.  For a board game, a state may contains the board position and whose turn The Co-Evolvability of Games in Coevolutionary Genetic Algorithms11

An Uninteresting Game: Needle-in-a-Haystack  Consider a game with two players, each player has N strategies and the payoff is: The Co-Evolvability of Games in Coevolutionary Genetic Algorithms12 P1 \ P N-1N 1(1,-1) (-1,1) 2(1,-1) (-1,1) 3(1,-1) (-1,1) N-1(1,-1)...(1,-1)(-1,1) N(1,-1) (0,0)

A Simple State Game The Co-Evolvability of Games in Coevolutionary Genetic Algorithms13  Moving a flag along a straight line, one unit a turn for both players, and terminates in t turns

Coevolutionary GA with RTS  Sample a random population  If terminating condition is not satisfied, repeat Generate a new population by crossover Evaluate the fitness of all individuals by playing games with a set of opponents in the population For each individual in the new population, find the nearest one in the old population and replace it if the fitness is higher. The Co-Evolvability of Games in Coevolutionary Genetic Algorithms14

Coevolution with Niching is Better The Co-Evolvability of Games in Coevolutionary Genetic Algorithms15

The Optimal Number of Opponents (1) The Co-Evolvability of Games in Coevolutionary Genetic Algorithms16

The Optimal Number of Opponents (2) The Co-Evolvability of Games in Coevolutionary Genetic Algorithms17

Evolving a Heuristic  In real-world, board games is large: The number of steps is proportional to the board size. The number of states is exponential to the number of steps. The number of strategies is still exponential to the number of states!  Thus we prefer to evolve a heuristic. The Co-Evolvability of Games in Coevolutionary Genetic Algorithms18

Less Co-Evolvable Games  A perfect strategy need to “remember” the decision at an exponential number of states. In the worst case, we need to perform an exponential number of tournaments.  A perfect heuristic may not evolvable due to the above large number. The Co-Evolvability of Games in Coevolutionary Genetic Algorithms19

Bit-Flipping Game The Co-Evolvability of Games in Coevolutionary Genetic Algorithms20

The Required Population Size The Co-Evolvability of Games in Coevolutionary Genetic Algorithms21

Conclusions  The mixed strategy games is not co- evolvable  Niching is a helpful technique to pure strategy games.  The optimal number of opponents used to evaluate a strategy exists.  The existence of less co-evolvable pure strategy games. The Co-Evolvability of Games in Coevolutionary Genetic Algorithms22