Machine Learning in Computer Games Marc Ponsen 11/29/04

2 Game AI: The Last Frontier “Progress in graphics and sound has slowed in recent years… Now, more than ever, good game play is at the forefront and AI is one of the most critical components (Rabin 2004).” “… Interactive computer games provide a rich environment for incremental research on human- level AI… Populating games with realistic, human- level characters will lead to fun, challenging games with great game play (Laird 2000).”

3 Why Learning of Game AI? The process of learning in games generally implies the adaptation of behavior for opponent players in order to improve performance Self-correction  Automatically fixing exploits Creativity  Responding intelligently to new situations Scalability  Better entertainment for strong players  Better entertainment for weak players

4 Online – during gameplay  Adapt to player tactics  Avoid repetition of mistakes  Requirements: computationally cheap, effective, robust, fast learning (Spronck 2004) Offline - before the game is released  Devise new tactics  Discover exploits Offline vs. Online Learning

5 Machine Learning Techniques

6 Overview Reinforcement Learning (RL)  Introduction to RL  RL application: Dynamic Scripting  Demo NeverWinter Nights © Evolutionary Algorithms (EA)  Introduction to EA  EA application: EA in Stratagus/Wargus  Demo Wargus

7 Introduction to Reinforcement Learning Techniques Framework based on punishments and rewards Maximize the frequency of rewards Minimize the frequency of punishments RL is popular research area because:  RL can solve wide variety of complex problems  RL will find close to optimal solution  RL learns as it interacts with the environment

8 How to provide Punishments & Rewards Weight adaptation function based on fitness score(s) Lookup tables Neural networks

9 When to provide Punishments & Rewards Reward when AI achieves objective or the opponent finds itself in a state where it can’t achieve its objective Reward when AI does something to increase the chance of achieving objective (guided rewards) Punish when opponent has reached objective Punish when AI does something to decrease the chance of achieving objective (guided negative rewards) Using both overall and guided (negative) rewards, speeds up the learning process

10 RL application: Dynamic Scripting Dynamic Scripting (DS) is an online learning technique inspired by RL Original implementation of DS (Spronck 2004) in the Computer RolePlaying Game NeverWinter Nights ©

11 Dynamic Scripting Rulebase A Rulebase B Script A Script B Combat generate script scripted control human control human control weight updates team controlled by human player team controlled by computer A B

12 Dynamic Scripting and Requirements Computationally Cheap - Script generation and weight updates once per encounter Effective - Rules are manually designed Robust - Reward/penalty system Fast Learning – Experiments showed that DS is able to adapt fast to an unchanging tactic

13 NeverWinter Nights © Demo

14 Dynamic Scripting in Wargus Different rulebases for different game states State transition on constructing a building that allows new units or new research

15 Rules in Rulebases 12 Build rules 9 Research rules 4 Economy rules 25 Combat rules AiNeed(AiBarracks) AiResearch(AiUpgradeArmor1) AiNeed(AiWorker) AiForce(1, {AiSoldier, 9}) AiWaitForce(1) AiAttackWithForce(1)

16 Script Generation State 1 Rulebase State n Rulebase Start Rule 1.1 Build Rule 1.x Rule n.1 State 20 Rulebase Rule Attack Loop Rule 20.19

17 Tactics Two `balanced’ tactics  Small Balanced Land Attack (SBLA)  Large Balanced Land Attack (LBLA) Two `rush’ tactics  Soldier Rush (SR)  Knight Rush (KR)

18 Dynamic Scripting Test Dynamic player (using dynamic scripting) plays 100 consecutive games against static player Randomisation Turning Point (RTP): First game that dynamic player outperforms static player with 90% probability according to a randomisation test (Cohen, 1995)

19 Dynamic Scripting RTP Results TacticTestsLowHighAvg.Med.>100Won SBLA LBLA SR KR Dynamic Scripting WORKS! It adapts efficiently against SBLA and LBLA Dynamic Scripting is unable to adapt to the optimized tactics SR and KR

20 Improve Domain-Knowledge Reliability Successful adaptive game AI is based invariably on domain knowledge of the game it is used in Step 1: Use EA to evolve new tactics to deal with problematic static opponents Step 2: Translate evolved tactics to improved domain knowledge (rulebase revision)

21 Introduction to Evolutionary Algorithms Idea: Biological analogy on how populations of species evolve over generations Step 1: start with a population (each member is a candidate solution) … Step 2: Create the next generation by considering evolutionary operations on the population from the previous generation (genetic operators such as mutation or crossover) and a fitness function (only the more fit get to contribute to the next generation) … Continue the process until a certain condition is reached

22 Chromosome AiForce(2, {AiSoldier, 5}) AiForceRole(“defend”)

23 Genetic Operators State Crossover Rule Replace Mutation Rule Biased Mutation Randomization

24 Wargus Demo

25 Evolution Results TacticTestsLowHighAvg. SR KR EA was successful in rapidly discovering tactics able to defeat both rush tactics

26 Analysis of Counter-Tactics Counter `soldier rush’ with (enhanced) `soldier rush’ Against `knight’s rush’  Expand territory when defendable  Quickly develop strong units  Use catapults

27 Rulebase Improvement New rules  Anti-SR: create strong soldiers and attack  Anti-KR: if a `knights’ state can be reached, do so and start building knights  Defendable expansion: build defensive force then expand Changes  Adjustment of parameters of several combat rules

28 Improved Rulebase Results TacticTestsLowHighAvg.Med.>100Won SBLA LBLA SR KR Enhanced performance against ALL tactics!

29 Conclusions Dynamic scripting can be successfully implemented in (commercial) CRPG & RTS games Offline evolutionary learning can design counter-tactics for RTS games Evolved counter-tactics can be used to improve domain knowledge used by adaptive game AI

30 References Spronck, P., I. Sprinkhuizen-Kuyper, and E. Postma. 2004a. “Online Adaptation of Game Opponent AI with Dynamic Scripting.” Sweetser, P. (2002). Current AI in Games: A review Manslow. “Learning and Adaptation” & “Using reinforcement learning to Solve AI Control Problems” Laird, J.E “Human-Level AI's Killer Application: Computer Game AI.” Ponsen, M. Spronck. P “Improving Adaptive AI with Evolutionary Learning”