Hierarchical Plan Representations for Encoding Strategic Game AI Hai Hoang Stephen Lee-Urban Héctor Muñoz-Avila Lehigh University

Outline Motivation Related Work Background  Unreal Tournament © (UT)  Hierarchical task Network (HTN) Planning Encoding Strategic Game AI in HTNs  Coordinating teams of UT Bots using HTNs  Video Using Processes-Based Languages to Encode HTNs Final Remarks  Future work

Motivation: FSM vs. Planning Planning Operators Patrol  Preconditions: No Monster  Effects: patrolled Fight  Preconditions: Monster in sight  Effects: No Monster PatrolFight Monster In Sight No Monster FSM: A resulting plan: Patrol patrolled Fight No Monster Monster in sight Neither is more powerful than the other one

But Planning Gives More Flexibility “Separates implementation from data” --- Orkin inference knowledge Planning Operators Patrol  Preconditions: No Monster  Effects: patrolled Fight  Preconditions: Monster in sight  Effects: No Monster … Many potential plans: PatrolFight PatrolFight PatrolFight PatrolFight PatrolFight …

Automated Planning: Uses in Games AI paradigm used to implement opponent’s behavior in FEAR (see previous class) Automated Planning is used to gain flexibility in designing Game AI behavior  An action: Attack positioned Weapon ready Enemy attacked preconditions effects

Example Attack positioned Weapon is ready Enemy attacked Stay-put positioned Take cover positioned Prepare Weapon Ammo loaded Holding weapon Weapon is ready How many plans can we generate to achieve Enemy attacked? FEAR defines 30+ such actions. So could generate many plans. How else could we use to create an opponent AI? FSMs (Project 1)

Optimization: Representation Attack positioned Weapon is ready Enemy attacked Declared as a variable in standard C++: Boolean EnemyAttacked; Declared as a C++ procedure: Action Attack() { if (positioned && WeaponISReady) EnemyAttacked = true; } Discussion points: Complexity versus an standard AI planner So why AI planners do not adopt this?

Optimization: Search A C B ABC AC B C B A B A C B A C BC A C A B A C B B C A AB C A B C A B C ( Michael Moll )

Motivation for using HTN planning versus Standard (STRIPS) Planning Goal-Oriented Action Planning (Orkin, 2004, 2005)  characters can find alternate solutions to situations encountered in the game  characters can handle dependencies that may not have been thought of at development time HTNs provide distinct advantages over “STRIPS” planning, on which GOAP is based Key questions:  What kinds of Game AI knowledge to represent in HTNs?  How to reason in dynamic game environments?  How to encode the HTNs Goals & operators Game developers should Learn LISP?

Related Work: FPS and AI Games have the potential to become testbed for different AI algorithms (Laird & van Lent, 1999) FPS-based system developed and used for actual training of military personnel (Laird & Duchim, 2000) Soar Bots for UT using the Soar Architecture (Laird & Duchim, 2000) GameBots: A group at USC-ISI and CMU jointly developed the Gamebot project to create an API for other clients to connect to the UT server.

Related Work: HTNs and Game AI Hierarchical planning can be useful in games (Wallace, 2003) Bridge Baron game uses HTN ideas (Smith et al., 1998). Hierarchical FSMs are an extension of FSMs in which states can expand into their own sub-FSMs (Houlette & Fu, 2003) HTNs used for dialog generation (Cavazza & Charles, 2005)

Unreal Tournament© Online FPS developed by Epic Games Inc. Various gameplay modes including team deathmatch and capture the flag Provides a client-server architecture for controlling bots High-level script language: UnrealScript (Sweeney, 2004) Project 2

Client-Server Architecture for UT Bots The UT server provides sensory information about events in the UT world and controls all the gameplay events. Event-driven: The client program uses this information to decide commands controlling the behavior of a Bot and passes them to the server. Two types of messages:  Asynchronous: special events  Synchronous: regular intervals Variants: Enhanced server (USC/ISI), Java Bots (CMU), Soar Bots (U. of Michigan).

Hierarchical Task Network (HTN) Planning Complex tasks are decomposed into simpler tasks. Seeks to decompose compound tasks into primitive tasks, which define actions changing the world The knowledge artifacts indicating how to decompose task into subtasks are called methods task t

Why HTN Planning (1)? HTN planning is provable more expressive than STRIPS planning (Erol et al, 1994) Even for those domains in which HTN representations can be translated into STRIPS representations, a much larger number of STRIPS operators is required to represent the HTN methods (Lotem & Nau, 2000) Fast planning through domain-configurable HTN planners (SHOP system)

Why HTN Planning (2)? Potential for efficiency gains: Rather than searching through possible states and actions a top-down task decomposition process is performed that may prune the search space substantially A C B ABC AC B C B A B A C B A C BC A C A B A C B B C A AB C A B C A B C Achieve 3-tower Achieve 2-tower Pile 3 rd block

Why HTN Planning (3)? Hierarchical planning is natural in many domains  Military planning Tactical Strategic Theater CINC JCS / NCA Strategic National JTF Operational

Application: UT Domination Games A number of fixed domination locations. When a team member steps into one of these locations, the status of the location changes to be under the control of his/her team. The team gets a point for every five seconds that each domination location remains under the control of that team. The game is won by the first team that gets a pre-specified amount of points.

Idea Use of hierarchical task network (HTN) planning techniques to laid out a grand strategy to accomplish gaming tasks Use standard event-driven programming allowing the bots to react in highly dynamic environments while contributing to the grand task UT task: Domination Strategy: secure most locations UT action: move Bot1 to location B

Technical Difficulties of using an HTN Planner (1) UT task: Domination Strategy: secure most locations UT action: move Bot1 to location B 1.Declarative language for expressing conditions 2. Dealing with contingencies while Bots are executing actions Solution to 1: use built- in predicates to represent conditions Solution to 2: use Java Bots to represent actions (event-driven)

Data Flow to generate HTNs (i.e., HTN Planning) task compound? Yes Select applicable method UT Server Method library Evaluate method’s preconditions Selected method’s subtasks No Execute action UT Bot …

Video

Empirical Results It is possible to encode strategies that coordinate teams of bots in FPS games using HTNs

Using Processes to Encode Game AI Think-Act Cycle Shoot SomethingPick up a HealthpackGo Somewhere Else Call Shoot Operator Ask Decision System: Where Do I Go? Ask Decision System: Where Do I Go? Call Pickup Operator

TIELT Project: Middleware between Games and AI Decision Systems Decision System Library Decision System Library Reasoning System Reasoning System Learning Module... Learning Module Reasoning System Reasoning System Learning Module... Learning Module Decision System Decision System Learning Module... Learning Module TIELT User TIELT’s Internal Communication Modules TIELT’s KB Editors TIELT’s KB Editors Selected/Developed Knowledge Bases Game Player(s) TIELT’s User Interface Evaluation Interface TIELT User Selected Game Engine Selected Game Engine Selected Decision System Selected Decision System Raw State Processed State DecisionAction Knowledge Base Libraries Knowledge Base Libraries Learned Knowledge (inspectable) Prediction Interface Coordination Interface Advice Interface Game Model Agent Description Game Interface Model Decision System Interface Model Experiment Methodology GIM DSIM GM AD EM Game Engine Library Game Engine Library Stratagus Full Spectrum Command EE2

Result: Processes can Be Translated into HTNs TIELTHTN Iterations (while)Recursion Iterations (for)Convert to while, recursion Tasks: preconditions and effectsMethods + operators If-elseIf-elseMethod FunctionsParameters of task (or call to external code, or Horn clause) Set-statement= command PrioritiesIf-then-else

Final Remarks HTNs can be used to model effective team strategies for Unreal Tournament  (UT) bots. A grand strategy is laid out by the HTNs and event- driven programming allows the bots to react in this highly dynamic environment while contributing to the grand task. Processes languages (e.g., TMKLs) can be translated into HTNs

Other Work Reinforcement learning to learn adequate task decomposition (upcoming class) Case-based reasoning techniques to capture and reuse gaming strategies (upcoming class: Bryan) Experiment to measure performance of translated processes