Turn-Based Games Héctor Muñoz-Avila sources: Wikipedia.org Russell & Norvig AI Book; Chapter 5 (and slides) Jonathan Schaeffer’s AAW 05 presentation My own

Turn-Based Strategy Games Early strategy games was dominated by turn- based games Derivate from board games Chess The Battle for Normandy (1982) Nato Division Commanders (1985) Turn-based strategy: game flow is partitioned in turns or rounds. Turns separate analysis by the player from actions “harvest, build, destroy” in turns Two classes: Simultaneous Mini-turns

Turn-Based Games Continues to be A Popular Game Genre At least 3 sub-styles are very popular: –“Civilization”-style games Civilization V will be released soon –Fantasy-style (RPG) Heroes of Might and Magic series –Poker games Poker Academy

Some Historical Highlights 1952 Turing design a chess algorithm. Around the same time Claude Shannon also develop a chess program 1956 Maniac versus Human 1970 Hamurabi. A game about building an economy for a kingdom The Battle for Normandy (1982) 1987 Pirates! 1990 Civilization 1995 HoMM 1996 Civilization II The best game ever? … 2006 HoMM V 2011 Civ V

Coming back: How to Construct Good AI? Idea: Lets just use A* and define a good heuristic for the game  Search space: a bipartite tree  After all didn’t we use it with the 9-puzzle game? Problems with this idea:  Adversarial: we need to consider possible moves of our opponent (s)  Time limit: (think Chess)

Categories of Games from an Artificial Intelligence (AI) Perspective Categories were, in part, the result of pursuing to apply AI techniques to construct “better” computer-controlled opponents Depending on the categories, building such AI turns out to be easier or more difficult In the context of game design, provides another way to analyze games –Specifically a different way from the traditional classification of games by genre

Deterministic Games Every player’s action results in a single pre-determined state?  Yes: Deterministic game state action

Chance Games Every player’s action results in a single pre-determined state?  No: Chance game A B Action: A cast a damage spell on B OutcomeOutcome: B blocks spell with 20% chances If B does not block spell, then damage dealt to B is randomly choose between 25%-40% of player’s B health points Action state 1 state 2 … (We don’t know apriori which until action is performed)

Perfect Information Games Does the player knows all information about the current state of the game?  Yes: perfect information game

Imperfect Information Games Does the player knows all information about the current state of the game?  No: imperfect information game

Types of Adversarial TBGs (from AI perspective) Perfect information Imperfect information Deterministic Chance Chess, Go, rock- paper-scissors Warcraft 1 Chutes and Ladders Civilization, HoMM Bridge, Poker We are going to study solutions for perfect information, deterministic games, for groundbreaking research including chance elements and imperfect information come to invited lecture on Thursday STEPS 101, 4PM

How Well does Computer Play Deterministic, Perfect Information Games?

Game tree (2-player, deterministic, turns) Concepts: State: node in search space Operator: valid move Terminal test: game over Utility function: value for outcome of the game MAX: 1 st player, maximizing its own utility MIN: 2 nd player, minimizing Max’s utility

Minimax Finding perfect play for deterministic games Idea: choose move to position with highest minimax value = best achievable payoff against best play E.g., 2-play game:

Minimax algorithm

Properties of minimax Complete? Optimal? Time complexity? –b: branching factor –m: # moves in a game Yes (if tree is finite) Yes (against an optimal opponent) O(b m ) For chess, b ≈ 35, m ≈100 for "reasonable" games Therefore, exact solution is infeasible

Minimax algorithm with Imperfect Decisions evaluationFunction(state) Cutoff-test(state)

Evaluation Function –Is an estimate of the actual utility –Typically represented as a linear function: EF(state) = w 1 f 1 (state) + w 2 f 2 (state) + … + w n f n (state) –Example: Chess weight: Piece  Number  (w 1 ) Pawn  1  (w 2 ) Knight  3  (w 3 ) Bishop  3  (w 4 ) Rook  5  (w 5 ) Queen  9 Function; state  Number  f 1 = #(pawns,w)  #(pawns,b)  f 2 = #(knight,w)  #(knight,b)  f 3 = #(bishop,w)  #(bishop,b)  f 4 = #(rook,w)  #(rook,b)  f 5 = #(knight,w)  #(knight,b)

Example: Evaluation Function “all things been equal” White moves, Who is winning? Is this consistent with Evaluation function? Black Yes!

Evaluation Function (2) Obviously, the quality of the AI player depends on the evaluation function Conditions for evaluation functions:  If n is a terminal node,  Computing EF should not take long  EF should reflect chances of winning EF(n) = Utility(n) If EF(state) > 3 then is almost-certain that blacks win

Cutting Off Search When to cutoff minimax expansion? Potential problem with cutting off search: Horizon problem Solution:  Fixed depth limit  Iterative deepening until times runs out  Decision made by opponent is damaging but cannot be “seen” because of cutoff  Quiescent: further explore states that may have more variance in results

Example: Horizon Problem “all things been equal” White moves, Who is winning? Is this consistent with Evaluation function? Black No!

α-β pruning: Motivation A good program may search 1000 positions per second In a chess tournament, a player gets 150 seconds per move Therefore, the program can explore 150,000 positions per move With a branching factor of 34, this will mean a look ahead of 3 or 4 moves Facts:  4-turns ≈ human novice  8-turns ≈ typical PC, human master  12-turns ≈ Deep Blue, Kasparov How to look ahead more than 4 turns? Use α-β pruning

Example: Finding perfect play for deterministic games Idea: choose move to position with highest minimax value = best achievable payoff against best play E.g., 2-play game:

α-β pruning

α-β pruning example

Principle of α-β Prunning α is the value of the best (i.e., highest- value) choice found so far at any choice point along the path for max  If v  α, max will avoid it oTherefore, prune that branch β is the lowest-value found so far at any choice point along the path for min  If v  , min will avoid it oTherefore, prune that branch

The α-β algorithm

Properties of α-β Pruning preserves completeness and optimality of original minimax algorithm Good move ordering improves effectiveness of pruning With "perfect ordering," time complexity = O(b m/2 ) Therefore, doubles depth of search Used in PC games today (9 moves look-ahead, Grand Master level)

Note The next 5 slides are from David W. Aha (NRL) presentation at Lehigh University in Fall’04 Claim: search space in Civilization is much larger than Chess

Example Game: FreeCiv (Chance, adversarial, imperfect information game) Civilization II  (MicroProse) Civilization II  (1996-): 850K+ copies sold –PC Gamer: Game of the Year Award winner –Many other awards Civilization  series (1991-): Introduced the civilization-based game genre Civilization II  (1996-): 850K+ copies sold –PC Gamer: Game of the Year Award winner –Many other awards Civilization  series (1991-): Introduced the civilization-based game genre FreeCiv (Civ II clone) Open source freeware Discrete strategy game Goal: Defeat opponents, or build a spaceship Resource management –Economy, diplomacy, science, cities, buildings, world wonders –Units (e.g., for combat) Up to 7 opponent civs Partial observability Open source freeware Discrete strategy game Goal: Defeat opponents, or build a spaceship Resource management –Economy, diplomacy, science, cities, buildings, world wonders –Units (e.g., for combat) Up to 7 opponent civs Partial observability

FreeCiv Scenario General description Game initialization: Your only unit, a “settler”, is placed randomly on a random world (see Game Options below). Players cyclically alternate play Objective: Obtain highest score, conquer all opponents, or build first spaceship Scoring: “Basic” goal is to obtain 1000 points. Game options affect the score. –Citizens: 2 pts per happy citizen, 1 per content citizen –Advances: 20 pts per World Wonder, 5 per “futuristic” advance –Peace: 3 pts per turn of world peace (no wars or combat) –Pollution: -10pts per square currently polluted Top-level tasks (to achieve a high score): –Develop an economy –Increase population –Pursue research advances –Opponent interactions: Diplomacy and defense/combat Game initialization: Your only unit, a “settler”, is placed randomly on a random world (see Game Options below). Players cyclically alternate play Objective: Obtain highest score, conquer all opponents, or build first spaceship Scoring: “Basic” goal is to obtain 1000 points. Game options affect the score. –Citizens: 2 pts per happy citizen, 1 per content citizen –Advances: 20 pts per World Wonder, 5 per “futuristic” advance –Peace: 3 pts per turn of world peace (no wars or combat) –Pollution: -10pts per square currently polluted Top-level tasks (to achieve a high score): –Develop an economy –Increase population –Pursue research advances –Opponent interactions: Diplomacy and defense/combat Game OptionY1Y2Y3 World sizeSmallNormalLarge Difficulty levelWarlord (2/6)Prince (3/6)King (4/6) #Opponent civilizations557 Level of barbarian activityLowMediumHigh

FreeCiv Concepts Concepts in an Initial Knowledge Base Resources: Collection and use oFood, production, trade (money) Terrain: oResources gained per turn oMovement requirements Units: oType (Military, trade, diplomatic, settlers, explorers) oHealth oCombat: Offense & defense oMovement constraints (e.g., Land, sea, air) Government Types (e.g., anarchy, despotism, monarchy, democracy) Research network: Identifies constraints on what can be studied at any time Buildings (e.g., cost, capabilities) Cities oPopulation Growth oHappiness oPollution Civilizations (e.g., military strength, aggressiveness, finances, cities, units) Diplomatic states & negotiations Resources: Collection and use oFood, production, trade (money) Terrain: oResources gained per turn oMovement requirements Units: oType (Military, trade, diplomatic, settlers, explorers) oHealth oCombat: Offense & defense oMovement constraints (e.g., Land, sea, air) Government Types (e.g., anarchy, despotism, monarchy, democracy) Research network: Identifies constraints on what can be studied at any time Buildings (e.g., cost, capabilities) Cities oPopulation Growth oHappiness oPollution Civilizations (e.g., military strength, aggressiveness, finances, cities, units) Diplomatic states & negotiations

FreeCiv Decisions Civilization decisions Choice of government type (e.g., democracy) Distribution of income devoted to research, entertainment, and wealth goals Strategic decisions affecting other decisions (e.g., coordinated unit movement for trade) Choice of government type (e.g., democracy) Distribution of income devoted to research, entertainment, and wealth goals Strategic decisions affecting other decisions (e.g., coordinated unit movement for trade) City decisions Unit decisions Diplomacy decisions Production choice (i.e., what to create, including city buildings and units) Citizen roles (e.g., laborers, entertainers, or specialists), and laborer placement –Note: Locations vary in their terrain, which generate different amounts of food, income, and production capability Production choice (i.e., what to create, including city buildings and units) Citizen roles (e.g., laborers, entertainers, or specialists), and laborer placement –Note: Locations vary in their terrain, which generate different amounts of food, income, and production capability Task (e.g., where to build a city, whether/where to engage in combat, espionage) Movement Task (e.g., where to build a city, whether/where to engage in combat, espionage) Movement Whether to sign a proffered peace treaty with another civilization Whether to offer a gift Whether to sign a proffered peace treaty with another civilization Whether to offer a gift

FreeCiv CP Decision Space Variables Civilization-wide variables oN: Number of civilizations encountered oD: Number of diplomatic states (that you can have with an opponent) oG: Number of government types available to you oR: Number of research advances that can be pursued oI: Number of partitions of income into entertainment, money, & research U: #Units oL: Number of locations a unit can move to in a turn C: #Cities oZ: Number of citizens per city oS: Citizen status (i.e., laborer, entertainer, doctor) oB: Number of choices for city production Civilization-wide variables oN: Number of civilizations encountered oD: Number of diplomatic states (that you can have with an opponent) oG: Number of government types available to you oR: Number of research advances that can be pursued oI: Number of partitions of income into entertainment, money, & research U: #Units oL: Number of locations a unit can move to in a turn C: #Cities oZ: Number of citizens per city oS: Citizen status (i.e., laborer, entertainer, doctor) oB: Number of choices for city production Decision complexity per turn (for a typical game state) O(D N GRI*L U *(S Z B) C ) ; this ignores both other variables and domain knowledge oThis becomes large with the number of units and cities oExample: N=3; D=5; G=3; R=4; I=10; U=25; L=4; C=8; Z=10; S=3; B=10 oSize of decision space (i.e., possible next states): 2.5*10 65 (in one turn!) oComparison: Decision space of chess per turn is well below 140 (e.g., 20 at first move) O(D N GRI*L U *(S Z B) C ) ; this ignores both other variables and domain knowledge oThis becomes large with the number of units and cities oExample: N=3; D=5; G=3; R=4; I=10; U=25; L=4; C=8; Z=10; S=3; B=10 oSize of decision space (i.e., possible next states): 2.5*10 65 (in one turn!) oComparison: Decision space of chess per turn is well below 140 (e.g., 20 at first move)

Open Discussion: Learning versus Exhaustive Search Learns from experience Learns same solution versus minimax “rational” opponent Explores all possibilities Generates best solution versus “rational” opponent