Strategic Planning for Unreal Tournament© Bots Héctor Muñoz-Avila Todd Fisher Department of Computer Science and Engineering Lehigh University USA Héctor Muñoz-Avila Todd Fisher Department of Computer Science and Engineering Lehigh University USA

Overview Motivation Unreal Tournament© (UT) HTN Planning  Why HTN Planning Related Work HTN Encodings of UT Bot Strategies Built-in predicates and Java Bots Final Remarks Motivation Unreal Tournament© (UT) HTN Planning  Why HTN Planning Related Work HTN Encodings of UT Bot Strategies Built-in predicates and Java Bots Final Remarks

Some research issues on reactive environments… Plan generation versus plan execution Case-Based Planning  Fast retrieval of adequate plans  Real-time plan adaptation Learning  Learning retrieval criteria  Learning domain knowledge  … Some research issues on reactive environments… Plan generation versus plan execution Case-Based Planning  Fast retrieval of adequate plans  Real-time plan adaptation Learning  Learning retrieval criteria  Learning domain knowledge  … Motivation # 1: Test Environment for Reactive Planning Lack of adequate testing environment

Motivation # 2: Declarative Encoding of Gaming Strategies Many games provide scripting languages (e.g., Lua scripting language) Games can be useful to teach programming concepts (e.g., IBM’s robocode) Is it possible to create a declarative language for encoding gaming strategies?  RBSs are an alternative (but can be messy)  How about for controlling teams of Bots? Avatar controlled by a computer program

Unreal Tournament© Online FPS developed by Epic Games Inc. Various gameplay modes including team deathmatch and capture the flag Provides a client-server architecture for controlling bots High-level script language: UnrealScript (Sweeney, 2004)

Hierarchical Task Network (HTN) Planning Principle: complex tasks are decomposed into simpler tasks. The goal is to decompose compound tasks into primitive tasks, which define actions changing the world. Fly(Newark, BWI) Travel(BWI, NRL) Travel(Lehigh,Newark Airport) alternative methods Travel from Lehigh to NRL Travel by car Enough money for gasoline Roads are passableSeats available Travel by plane Enough money for air fare available Drive(Lehigh, Newark Park) Bus(Newark Park,airport) Taxi(BWI, NRL) Travel(Lehigh,NRL)

Why HTN Planning? HTN planning is provable more expressive than STRIPS planning (Erol et al, 1994) Hierarchical planning is natural in many domains Tactical Strategic Theater CINC JCS / NCA Strategic National JTF e.g., military planning Operational Can be very fast!

Related Work Hierarchical planning can be useful in games (Wallace, 2003) Bridge Baron game uses HTN ideas (Smith et al., 1998). Games have the potential to become testbed for different AI algorithms (Laird & van Lent, 1999). Soar Bots for UT using the Soar Architecture (Laird & Duchim, 2000) Tielt: testbed for investigating and evaluating learning techniques (Aha & Molineaux, 2004) Hierarchical planning can be useful in games (Wallace, 2003) Bridge Baron game uses HTN ideas (Smith et al., 1998). Games have the potential to become testbed for different AI algorithms (Laird & van Lent, 1999). Soar Bots for UT using the Soar Architecture (Laird & Duchim, 2000) Tielt: testbed for investigating and evaluating learning techniques (Aha & Molineaux, 2004)

Client-Server Architecture for UT Bots The UT server provides sensory information about events in the UT world and controls all the gameplay events. Event-driven: The client program uses this information to decide commands controlling the behavior of a Bot and passes them to the server. Two types of messages :  Asynchronous: special events  Synchronous: regular intervals Variants: Enhanced server (UC/ISI), Java Bots (CMU), Soar Bots (U. of Michigan). The UT server provides sensory information about events in the UT world and controls all the gameplay events. Event-driven: The client program uses this information to decide commands controlling the behavior of a Bot and passes them to the server. Two types of messages :  Asynchronous: special events  Synchronous: regular intervals Variants: Enhanced server (UC/ISI), Java Bots (CMU), Soar Bots (U. of Michigan).

HTN for Controlling Team of Bots Idea: Use methods to encode gaming strategies Use operators to encode gaming actions Idea: Use methods to encode gaming strategies Use operators to encode gaming actions UT task: Domination Strategy: secure most locations UT action: move Bot1 to location B A B C

Technical Difficulties and Solutions UT task: Domination Strategy: secure most locations UT action: move Bot1 to location B 1.Declarative language for expressing conditions 2. Dealing with contingencies while Bots are executing actions Solution to 1: use built- in predicates to represent conditions Solution to 2: use Java Bots to represent actions (event-driven)

Example A B C HTN Method Head: Domination Preconditions: LocsGeographTogether(P,RP) Divide3Groups(T1,T2,T3) Subtasks: CoverLocations(T1,P) PatrolLocations (T2,P) HarrassLocations(T3,RP) P RP The HTNs are represented in XML

Strategy Change vs. Strategy Modification We are exploring 2 alternatives:  Select alternative strategy when similarity of current strategy falls below a certain threshold  Modify strategy using plan adaptation techniques

On-going/Short-Term Future Directions Selection of alternative strategy modification versus strategy modification Support the Tielt testbed:  Use HTNs and process models (TMKL) to encode game model  Learn conditions for retrieving strategies Selection of alternative strategy modification versus strategy modification Support the Tielt testbed:  Use HTNs and process models (TMKL) to encode game model  Learn conditions for retrieving strategies

Final Remarks Our approach can be used to accomplish two seemingly contradictory goals:  The Bot needs to react quickly in a highly dynamic environment.  The Bot must contribute to the grand strategy to win the game.  A grand strategy is laid out in the HTNs and event-driven programming allows the Bots to react in highly dynamic environments while contributing to the grand task. Our approach can be used to accomplish two seemingly contradictory goals:  The Bot needs to react quickly in a highly dynamic environment.  The Bot must contribute to the grand strategy to win the game.  A grand strategy is laid out in the HTNs and event-driven programming allows the Bots to react in highly dynamic environments while contributing to the grand task.