1. Artificial Intelligence in Game Design N-Grams and Decision Tree Learning

2. Predicting Player Actions Type of games where works best: Player has choice between few possible actions –Attack left –Attack right Character can take simultaneous counteraction –Each player action has “correct” counteraction Attack left  defend left Attack right  defend right Goal: Character should learn to “anticipate” current player action based on past actions

3. Probabilistic Approach Keep track of last n actions taken by player –Size of n is “window” of character memory Compute probability of next player action based on these –Base decision about counteraction on that probability Example: Last 10 player actions L L R L R L L L R L Estimated probabilities of next action: Left attack70% Right attack30%

4. Probabilistic Actions Simple majority approach –Since left attack has highest probability, defend left –Problem: Character will take same action for long periods Too predictable! –Example: Player’s next 4 attacks are from right Character will still choose defend left since left attacks still have highest probability Character looks very stupid! player: L L R L R L L L R L R R R R character: L L L L Left attack still majority player action in last 10 moves

5. Probabilistic Actions Probabilistic Approach: Choose action with same probability as corresponding player action Biased towards recent player actions Far less predictable –Player may notice that character is changing tactics without being aware of how this is done Left attack70% Right attack30% Left defend70% Right defend30%

6. Window Size Key question: What is good value for n? L L R L R L L L R L R R L Can be too small (n = 2, for example) L L R L R L L L R L R R L Can be too large (n = 20, for example) LLLLLLLLLLLLRRRRRRRR No best solution –Will need to experiment Size of “window” used to determine probabilities Character has no “memory” of past player actions Left defend50% Right defend50% Too slow to react to changes in player tactics Left defend60% Right defend40%

7. N-Grams Conditional probabilities based on sequence of user actions Example: –“Last two player attacks were left and then right” –“What has player done next after last times the attacked left then right?” After last 4 left then right attacks: –Attacked right 3 times –Attacked left 1 time Conclusion: Player has 75% chance of attacking right next

8. N-Grams Example Example: –Window of memory = last 12 actions –Base decision on last two actions taken by player (past sequence length = 2) Goal: Determine what action player is likely to take next given last two actions L R –Previous actions: L L R L R R L L R L L R ? –Previous cases of L R: Followed by L twice Followed by R once Left attack67% Right attack33%

9. N-Grams and Sequence Size Number of statistics to keep grow exponentially with length of past sequences –Number of possible actions = a –Past sequences length = L –Number of possible configurations of past sequences = a L L must be small (no more than 2 or 3)

10. Storing N-Grams Algorithm: –Keep statistics for all possible action strings of length L Based on “window” of past actions Example: L L L L R R L L R L L R Previous action string Instances of next action Probability next action is L Probability next action is R L L L R R R40%60% L R 50% R LL 100%0% R L100%0%

11. N-Grams and Updating When player takes new action –Add instance for that action and update statistics –Remove oldest action from list and update statistics Move “window” Example: L L L L R R L L R L L R L New action Now outside “window” Previous action string Instances of next action Probability next action is L Probability next action is R L L L R R R25%75% L RL R L67%33% R LL 100%0% R L100%0%

12. Decision Trees Simple tree traversal Node = question Branch to follow = answer Leaf = final action to take Can create dynamically from a set of examples Hit points < 5? yes no Obstacle between myself and player? yes hide no run Within one unit of player? yes attack no Path to player clear? yes run towards player no run sideways

13. Decision Tree Learning Based on a set of training examples –Attribute values about current situation From world or from character state –Target action character should take on basis of those attribute values Example: Estimating driving time –Hour of departure: 8, 9, or 10 –Weather: sunny, cloudy or rainy –Accident on route: yes or no –Stalled car on route: yes or no –Commute time: short, medium, or long attributes desired action

14. Training Examples ExampleHourWeatherAccidentStallCommute 18sunnyno long 28cloudynoyeslong 310sunnyno short 49rainyyesnolong 59sunnyyes long 610sunnyno short 710cloudyno short 89rainyno medium 99sunnyyesnolong 10 cloudyyes long 1110rainyno short 128cloudyyesnolong 139sunnyno medium

15. Decision Tree Building node BuildTree(examples[ ] E) { if (all examples in E have same target action T) { return new leaf with action T } else { choose “best” attribute A to create branches for examples E set question at this node to A for (all possible values V for attribute A) { create branch with value V E V = all examples in E that have value V for attribute A attach BuildTree(E V ) to that branch } } }

16. Decision Tree Building Start tree by using BuildTree(all examples) to create root Example: Suppose Hour were chosen at root: Hour Examples 1 – 13 4 short, 2 medium, 7 long another question 8 Examples 3, 6, 7, 10, 11 4 short, 0 medium, 1 long another question 10 Examples 4, 5, 8, 9, 13 0 short, 2 medium, 3 long 9 Examples 1, 2, 12 0 short, 0 medium, 3 long Long

17. ID3 Decision Tree Learning Choose attribute with highest information gain Based on definition of entropy from information theory Entropy over examples E with target actions T: Entropy(E) = - | E T | log 2 | E T | T | E | | E | Example: entropy for training set = -(4/13) log 2 (4/13) -(2/13) log 2 (2/13) -(7/13) log 2 (7/13) = 1.42 4 short examples 2 medium examples 7 medium examples 

18. ID3 Decision Tree Learning Expected entropy remaining after attribute A used = –Sum of entropies of child nodes down branches V –Weighted by percentage of examples down that branch | E V | Entropy (E V ) V | E | Information gain for attribute A = that value subtracted from original entropy 

19. AttributeEntropyInformation Gain hour0.650.77 weather1.290.13 accident0.920.50 stall1.170.25 Best attribute at root

20. Using ID3 in Games ID3 efficient enough for on-line learning –Execution time proportional to number of examples Best applied to games if: –Few possible actions for NPC to choose from (attack, retreat, defend) –Key attributes have few possible values (or continuous values partitioned into predefined ranges) –Have easy way to determine desired actions NPC should take based on significant number of player actions May require tweaking rules of game!

21. Black and White Game Player given “creature” at beginning of game Creature “trained” by player by observing player actions in different situations –What other armies to attack in what circumstances Later in game (after creature grows), creature takes those same actions

22. Black and White Game Sample training set: ExampleAllegianceDefenseTribeAttack 1friendlyweakCelticno 2enemyweakCelticyes 3friendlystrongNorseno 4enemystrongNorseno 5friendlyweakGreekno 6enemymediumGreekyes 7enemystrongGreekno 8enemymediumAztecyes 9friendlyweakAztecno

23. Black and White Game Tree created from examples: Allegiance Defense friendly No attack enemy weak Attack strong No attack medium Attack

24. Black and White Game Nature of game suited to decision tree learning: –Large number of examples created by player actions –Known target actions based on player actions –Small number of possible actions, attribute values Game specifically designed around algorithm!