1. An Introduction to ABL-Wargus Ashwin Ram / Michael Mateas / Charles Isbell

2. GILA Kickoff 30 May 06 2 The Wargus Domain Real-time strategy (RTS) –No turns, continuous time –Massive number of games states and actions Open-source clone of Warcraft II –Built on Stratagus game engine –Written in C/C++

3. GILA Kickoff 30 May 06 3 Wargus As A Testbed Actively used in AI research –Wargus Challenge –Case-Based Plan Selection (Aha et al) Commonly used abstraction layer: Lua (TIELT) –Build, Train, Upgrade, AttackWithForce, Repair, … Better: Wargus C abstraction layer –Enables tactical level decisions –Supports Fog of War –Explicit control of resource collection/allocation –Unit level control: Stop, StandGround, CastSpell, … –Building level control: Research, Upgrade, …

4. GILA Kickoff 30 May 06 4 Step 1: Planning. HTN Planner produces a hierarchical plan (task network). Screenshots show the scenario as the plan is executed. Task decomposition Informs Example GILA Scenario Base building in Wargus

5. GILA Kickoff 30 May 06 5 Enemy unit (catapult) has longer range - The defense tower network is ineffective Reasoning failure HTN task decomposition failure: cannot handle catapults New Learning Goal: How to beat catapults Blame Assignment Step 2: Meta-Reasoning. Plan failure results in blame assignment, which identifies reasoning failure and corresponding learning goals. Even the best plans can fail

6. GILA Kickoff 30 May 06 6 Learning from observation Barracks Defeat Catapult Patrol Footmen Build Footmen Learning Goal: defeat catapult CBR Learner - Uses observation trace to learn - Creates new case plan for killing catapults New Learned Case New defense strategy is integrated into full plan How do humans fight catapults? catapult1 12, 6 [cat1] [cat1] footman1 attacks [cat1] footman2 attacks [cat1] footman1 23 footman2 8 footman3 attacks [cat1] footman4 attacks [cat1] [cat1] 7 [cat1] 9... Observation Step 3: Goal-Driven Learning. Case-based learner “helps out” HTN planner by learning a case for the “knowledge gap”. Trace Capture

7. GILA Kickoff 30 May 06 7 From HTN Planner From CBR Learner Step 4: Integrated Reasoning. System can now handle long-range attackers by combining HTN Planning with CBR Planning. Adapted plan

8. GILA Kickoff 30 May 06 8 GILA-on-Wargus: Architecture Wargus Sensory-motor abstraction Wargus Agent Framework Agent Runtime Behavior Library Enemy Agent AI/Human Player Network Connection Socket communication GILA perceives world state and produces agent behaviors in “workflow execution language” Behaviors are executed in real-time Wargus world Enemy agents may execute behaviors concurrently over the network

9. GILA Kickoff 30 May 06 9 GILA-on-Wargus: Interface Wargus Enemy Agent AI/Human Player Network Connection ? Wargus Sensory-motor abstraction Agent Framework Agent Runtime Behavior Library Socket communication

10. GILA Kickoff 30 May 06 10 ABL: A Behavior Language Wargus Enemy Agent AI/Human Player Network Connection Scalable framework for AI agents in real-time, multi-agent domains. Wargus Sensory-motor abstraction ABL ABL Runtime Behavior Library Socket communication The interactive drama Façade contains 200,000+ lines of ABL code. Currently at >300,000 downloads. ABL has been interfaced with Unreal Tournament, Never Winter Nights, …

11. GILA Kickoff 30 May 06 11 ABL: A Behavior Language A reactive planning language for real-time, complex agents ABL features: Sequential & parallel behaviors Demons & continuously monitored conditions Multiple simultaneous goal pursuit with reactivity Joint goals and behaviors (joint intentions) Reflection (meta-behaviors) Sensory-motor abstraction Mateas & Stern (2004) Open question: Is ABL the “workflow execution language”?

12. GILA Kickoff 30 May 06 12 ABL Overview Active Behavior Tree Root behavior Goal 1 Goal 2 Seq. Behavior 1 Par. Behavior 2 MentalActGoal 3 Available for execution Behavior Library Behavior 1 Behavior 2 Behavior n Working Memory WME 1 WME 2 WME n Sensors Sensor 1 World

13. GILA Kickoff 30 May 06 13 Example Behaviors parallel behavior AttackEnemy() { context_condition { e = (UnitWME Visibility == VISIBLE) } act attack(e); subgoal KeepUnitsEngaged(); subgoal ConcentrateFire(e); } To attack the enemy: ● Wait for an enemy to come ● Send attack command ● Ensure units do not get stuck or wander ● Make range units concentrate fire sequential behavior HealUnit() { precondition { (UnitWME health < CRITICAL unitID::target location::tcoord) (UnitWME canHeal==true mana > 0 unitID::healer location::hcoord spellrange::range) (tcoord.distanceto(hcoord) < range) } act castspell(HEAL, healer, target); } To heal unit: ● Check for the corresponding preconditions for healing ● Cast healing spell

14. GILA Kickoff 30 May 06 14 sequential behavior ProtectBase (){ precondition { attackbeh = (GoalStepWME signature == “AttackEnemyBase()”) (BaseIsUnderAttack) } mental_act{ attackbeh.fail(); } ● We send an attacking force to invade enemy base ● En route, however, our own base comes under attack ● We cancel the invasion branch of the ABT so that forces can be freed to defend the base. Example Meta-Behaviors

15. GILA Kickoff 30 May 06 15 ABT Root behavior Par. behavior 2 Goal 2 Goal 3 Act 2 Seq. behavior 1 MentalAct 1 Act 2 Fail Seq. behavior 3 Seq. behavior 3 Fail Goal 3 Goal 2 Success Test Succeeds Mental Remove Seq. behavior 1 ABT (ABL Behavior Tree)

16. GILA Kickoff 30 May 06 16 Goals, Actions, Conditions Steps: subgoals, primitive acts, mental acts, wait –Waits are used with conditions to accomplish demons –All steps succeed or fail –Success and failure propagate up the ABT Continuously monitored conditions –Success tests – spontaneously succeeds a step if test is satisfied –Context conditions – spontaneously fails a behavior if test is satisfied –Makes behaviors immediately reactive to changes in the world

17. GILA Kickoff 30 May 06 17 Joint Goals And Behaviors Agent teams need to coordinate action with variable coupling Some approaches Coordinate through sensing (but plan recog. hard) Coordinate through sensing (but plan recog. hard) Explicitly communicate (but ad hoc) Explicitly communicate (but ad hoc) Build coordination into decision architecture Build coordination into decision architecture (but no fine-grained authorial control) (but no fine-grained authorial control) Architecture coordinates author-specified joint action

18. GILA Kickoff 30 May 06 18 Negotiation Agent 1 ’s ABT Root behavio r Goal 1 Joint Goal 2 Seq. Behavior 1 Joint Behavior 2 Menta l Agent 2 ’s ABT Root behavio r Goal 5 Seq. Behavior 5 Act Joint Goal 2 Joint Behavior 6 Intention to enter G 2

19. GILA Kickoff 30 May 06 19 Conflicting Intentions Agent 1 ’s ABTAgent 2 ’s ABTAgent 3 ’s ABT Root behavior Goal 1 Joint Goal 2 Succeeds Root behavior Goal 3 Seq. Behavior 1 Joint Goal 2 Joint Behavior 2 Act Fails Root behavior Goal 4 Seq. Behavior 3 Joint Goal 2 Joint Behavior 4 Act Suspends Resolution: intentions are precedence ordered MentalGoal 5 Problem: asynchronous agents enter conflicting states

20. GILA Kickoff 30 May 06 20 Inconsistent Subtree Execution ABT Root behavior Joint Goal 1 G2G2 Seq. Behavior 1 Joint Behavior 2 Joint Goal 2 Act Succeeds Resolution: freeze subtree Initiate exit intention at the subtree root Remove all leaf steps Deactivate all monitored conditions Negotiate removal of all joint goals Commit to exit intention at subtree root Problem: continuing execution leads to ABT inconsistencies

21. GILA Kickoff 30 May 06 21 Variably Coupled Multi-Mind Agent 1 ’s ABT Seq. Behavior 1 Root behavior Goal 1 Joint Goal 2 Joint Behavior 2 Act Agent 2 ’s ABT Root behavior Goal 3 Joint Goal 2 Joint Behavior 4 Mental Seq. Behavior 3 Effects propagate across ABTs MentalGoal 4 Effects propagate within ABTs Tunable spectrum between one-mind and many-minds

22. GILA Kickoff 30 May 06 22 A 2 BL: ABL + Machine Learning Turn machine learning into a language-level primitive –Learn local, behavior-specific policies Allow hierarchical mixtures of human-authored and machine- learned behaviors –Actions chosen by local policy may root entire reactive subtrees New language constructs for specifying reinforcement –Behaviors may be adaptive or non- adaptive Adaptive sequential behaviors –Hierarchical reinforcement learning (HRL) –Temporal decomposition of Q- function Adaptive collection behaviors –Modular reinforcement learning (MRL) –Concurrent decomposition of Q- function Case-based behavior generation –Case-based reasoning & learning (CBR) –Automatic creation of new ABL behaviors

23. GILA Kickoff 30 May 06 23 Example A 2 BL code adaptive sequential behavior AttackEnemy() { success_condition {!(EnemyWME health>0 visibility==VISIBLE)} reward { 10 if {(UnitWME player!=SELF id::theID) (DamageWME receiver==theID)} -10 if {(UnitWME player==SELF id::theID) (DamageWME receiver==theID)} } state { (ForceWME units::u location::coord) return (u, coord); } subgoal KeepUnitsEngaged(); subgoal ConcentrateFire(); subgoal MoveTowardsEnemy(); act attack(); subgoal Retreat(); subgoal AttackLongRange(); subgoal AttackShortRange(); }

24. GILA Kickoff 30 May 06 24 ABL-Wargus Communications Interface Wargus Enemy Agent AI/Human Player Network Connection Wargus Sensory-motor abstraction ABL ABL Runtime Behavior Library Socket communication ABL talks to Wargus via a Sensory-Motor System

25. GILA Kickoff 30 May 06 25 ABL-Wargus Communication Interface ProxyBot class implements low-level socket communication with Wargus Sensory-Motor System defines new actions and sensors Action classes define primitive actions to be performed on the Wargus side, eg. Attack, Repair, Follow, Dismiss, Cast Spell Actions call ProxyBot's corresponding functions to send messages to Wargus (via socket) WMEs (Working Memory Elements) are created corresponding to game state information received

26. GILA Kickoff 30 May 06 26 Example Sensors and WMEs EnemyInfoWME public void EnemyInfoWME extends WME{ public boolean getRace(){ return race;} public int getNumUnits() {return numUnits;} } EnemyInfoSensor public class EnemyInfoSensor extends WargusSensors { private void sense { EnemyInfoWME [] enemyinfo = proxy.GetAllEnemyInfo(); deleteAllOldWME(); addallEnemyInfoWME(enemyinfo);} private void addallEnemyInfoWME (EnemyInfoWME[] enemyinfo){ for(int i = 0; i < enemyinfo.length; i++) { BehavingEntity.getBehavingEntity().addWME(enemyinfo[i]); }

27. GILA Kickoff 30 May 06 27 Battle Management Example GILA faces attack by catapults and ogres. Catapults are destroying defense towers. GILA sends knights and archers to defend against attack.

28. GILA Kickoff 30 May 06 28 Battle Management Example (cont.) Knights encounter ogres and stop to attack. In the meantime, enemy catapults destroy defense towers. Unsupported units are quickly destroyed.

29. GILA Kickoff 30 May 06 29 GILA-Wargus Evaluation Use internal Wargus AI as opponent Show improvement via integrated learning –Baseline: Play against Wargus AI (w/wo learning) –Observational learning (human studies) –Integrated learning (lesion studies) Metrics –Percent of wins –Kill ratio –Time to accomplish objective