1. 1 Integrating Learning in Interactive Gaming Simulators David W. Aha 1 & Matthew Molineaux 2 1 Intelligent Decision Aids Group Navy Center for Applied Research in AI Naval Research Laboratory; Washington, DC 2 ITT Industries; AES Division; Alexandria, VA surname@aic.nrl.navy.mil AAAI’04 Workshop on Challenges in Game AI 25 July 2004

2. 2 Integrating Learning in Interactive Gaming Simulators Outline 1.Motivation: Learning in cognitive systems 2.Objectives: –Support empirical studies w/ simulators –Encourage studies that address industry & military M&S concerns 3.Design: TIELT functionality & components 4.Example: Knowledge base content 5.Status: Implementation, collaborations 6.Summary 1.Motivation: Learning in cognitive systems 2.Objectives: –Support empirical studies w/ simulators –Encourage studies that address industry & military M&S concerns 3.Design: TIELT functionality & components 4.Example: Knowledge base content 5.Status: Implementation, collaborations 6.Summary Thanks to our sponsor:

3. 3 Integrating Learning in Interactive Gaming Simulators Rough Anatomy of a Cognitive Agent External Environment Communication (language, gesture, image) Prediction, planning Deliberative Processes Reflective Processes Reactive Processes Perception Action STM SensorsEffectors Other reasoning LTM Concepts Sentences Cognitive Agent Affect Attention Learning (Brachman, 2003)

4. 4 Integrating Learning in Interactive Gaming Simulators Problem Status of Cognitive Learning Few deployed cognitive systems integrate techniques that exhibit rapid & enduring learning behavior on complex tasks –It’s costly to integrate & evaluate embedded learning techniques Few deployed cognitive systems integrate techniques that exhibit rapid & enduring learning behavior on complex tasks –It’s costly to integrate & evaluate embedded learning techniques Complication The ML research community has been focusing on: ¬Rapid: Knowledge poor algorithms ¬Enduring: Learning over a short time period ¬Embedded: Stand-alone evaluations The ML research community has been focusing on: ¬Rapid: Knowledge poor algorithms ¬Enduring: Learning over a short time period ¬Embedded: Stand-alone evaluations

5. 5 Integrating Learning in Interactive Gaming Simulators Wanted: A New Interface (thanks to W. Cohen, others) Supervised Learning ML System Database Interface (standard format) (e.g., UCI Repository) Reasoning System Supervised Learning ML System Database Interface (standard format) (e.g., UCI Repository) Reasoning System Supervised Learning ML System j Database i Interface (standard format) (e.g., UCI Repository of ML Databases) Reasoning System k Cognitive Learning Reasoning Modules World (Simulated/Real) Sensors ML Module Interface (standard API) ML Module (e.g., TIELT) Effectors Cognitive Learning Reasoning Modules World (Simulated/Real) Sensors ML Module Interface (standard API) ML Module (e.g., TIELT) Effectors Cognitive Learning Reasoning System k World i (Simulated/Real) Sensors ML Module Interface (standard API) ML Module ML Module j (e.g., TIELT) Effectors Testbed for Integrating and Evaluating Learning Techniques

6. 6 Integrating Learning in Interactive Gaming Simulators Objectives & Domain Objective Facilitate the evaluation of learning techniques in CogSys 1.Develop & transition TIELT to help reduce integration costs (time, $) 2.Support DARPA Challenge Problems on Cognitive Learning 3.Demonstrate research utility prior to approaching industry/military Facilitate the evaluation of learning techniques in CogSys 1.Develop & transition TIELT to help reduce integration costs (time, $) 2.Support DARPA Challenge Problems on Cognitive Learning 3.Demonstrate research utility prior to approaching industry/military Domain: Why interactive gaming simulators? 1.Available implementations (cheap to acquire & run) 2.Challenging problems for CogSys/ML research 3.Significant interest (military, industry, academia, funding, public) 1.Available implementations (cheap to acquire & run) 2.Challenging problems for CogSys/ML research 3.Significant interest (military, industry, academia, funding, public)

7. 7 Integrating Learning in Interactive Gaming Simulators TIELT Specification 1.Simplifies integration & evaluation! Learning-embedded reasoning systems & gaming simulators Inputs: 5 descriptions (simulator I/O, game model, learning & performance tasks, reasoning system I/O, & evaluation strategy) Free 2.Learning foci: Many Task (e.g., learn how to execute, or advise on, a task) Player (e.g., learn/predict a human player’s strategies) Game (e.g., learn/refine its objects, their relations, & behaviors) 3.Learning methods: Many types Supervised/unsupervised, immediate/delayed feedback, analytic, active/passive, online/offline, direct/indirect, automated/interactive Learning results should be available for inspection 4.Gaming simulators: Those with challenging learning tasks 5.Reuse: Provide access to libraries of simulators & reasoning systems Abstracts interface definitions from game & task models 1.Simplifies integration & evaluation! Learning-embedded reasoning systems & gaming simulators Inputs: 5 descriptions (simulator I/O, game model, learning & performance tasks, reasoning system I/O, & evaluation strategy) Free 2.Learning foci: Many Task (e.g., learn how to execute, or advise on, a task) Player (e.g., learn/predict a human player’s strategies) Game (e.g., learn/refine its objects, their relations, & behaviors) 3.Learning methods: Many types Supervised/unsupervised, immediate/delayed feedback, analytic, active/passive, online/offline, direct/indirect, automated/interactive Learning results should be available for inspection 4.Gaming simulators: Those with challenging learning tasks 5.Reuse: Provide access to libraries of simulators & reasoning systems Abstracts interface definitions from game & task models

8. 8 Integrating Learning in Interactive Gaming Simulators Distinguishing TIELT SystemFocus$Game Engine(s) Prominent Feature Reasoning Activity DirectIA (MASA) AI SDK FPS, RTS, etc. Behavior authoringSense-act, … SimBionic (SHAI) AI SDK FPS, etc.Behavior authoringSense-act, … FEARAI SDKQuake 2, etc.Behavior authoringSense-act, … RoboCupResearch Testbed RoboCupSoccer game playSense-act, coaching, etc. GameBotsResearch Testbed UT (FPS)UT game playSense-act ORTSResearch Testbed RTS gamesHack-free MM RTSSense-act, strategy TIELTResearch Testbed Several genres Experimentation for Learning Systems Sense-act, advice processing, prediction, model updating, etc. 1.Provides an interface for message-passing interfaces 2.Supports composable system-level interfaces

9. 9 Integrating Learning in Interactive Gaming Simulators TIELT’s Internal Communication Modules TIELT’s KB Editors TIELT’s KB Editors Selected/Developed Knowledge Bases Game Model Description Task Descriptions Game Interface Description Reasoning Interface Description Evaluation Methodology Description Game Player(s) Game Engine Library Game Engine Library Stratagus Full Spectrum Command TIELT’s User Interface Prediction Interface Evaluation Interface Coordination Interface Advice Interface TIELT User TIELT User Selected Game Engine Selected Game Engine Reasoning System Library Reasoning System Library Reasoning System Reasoning System Learning Module... Learning Module Reasoning System Reasoning System Learning Module... Learning Module Reasoning System Reasoning System Learning Module... Learning Module Selected Reasoning System Selected Reasoning System Learned Knowledge (inspectable) TIELT: Integration Architecture Knowledge Base Libraries Knowledge Base Libraries GID RID GMD TDs EMD......

10. 10 Integrating Learning in Interactive Gaming Simulators TIELT’s Knowledge Bases Game Model Description Task Descriptions Game Interface Description Reasoning Interface Description Evaluation Methodology Description Defines communication processes with the game engine Defines communication processes with the learning system Defines interpretation of the game e.g., initial state, classes, operators, behaviors (rules) Behaviors could be used to provide constraints on learning Defines the selected learning and performance tasks Selected from the game model description Defines the empirical evaluation to conduct

11. 11 Integrating Learning in Interactive Gaming Simulators TIELT’s Internal Communication Modules TIELT’s KB Editors TIELT’s KB Editors Selected/Developed Knowledge Bases Game Model Description Task Descriptions Game Interface Description Reasoning Interface Description Evaluation Methodology Description Game Player(s) Game Engine Library Game Engine Library Stratagus Full Spectrum Command TIELT’s User Interface Prediction Interface Evaluation Interface Coordination Interface Advice Interface TIELT User TIELT User Selected Game Engine Selected Game Engine Reasoning System Library Reasoning System Library Reasoning System Reasoning System Learning Module... Learning Module Reasoning System Reasoning System Learning Module... Learning Module Reasoning System Reasoning System Learning Module... Learning Module Selected Reasoning System Selected Reasoning System Learned Knowledge (inspectable) Example: Controlling a Game Character Knowledge Base Libraries Knowledge Base Libraries GID RID GMD TDs EMD Raw State Processed State DecisionAction

12. 12 Integrating Learning in Interactive Gaming Simulators TIELT’s Internal Communication Modules TIELT’s KB Editors TIELT’s KB Editors Selected/Developed Knowledge Bases Game Model Description Task Descriptions Game Interface Description Reasoning Interface Description Evaluation Methodology Description Game Player(s) Game Engine Library Game Engine Library Stratagus Full Spectrum Command TIELT’s User Interface Prediction Interface Evaluation Interface Coordination Interface Advice Interface TIELT User TIELT User Selected Game Engine Selected Game Engine Reasoning System Library Reasoning System Library Reasoning System Reasoning System Learning Module... Learning Module Reasoning System Reasoning System Learning Module... Learning Module Reasoning System Reasoning System Learning Module... Learning Module Selected Reasoning System Selected Reasoning System Learned Knowledge (inspectable) Example: Updating a Game Model Knowledge Base Libraries Knowledge Base Libraries GID RID GMD TDs EMD Raw State Processed State Edit

13. 13 Integrating Learning in Interactive Gaming Simulators Selected Game Engine Evaluation Editor Game Interface Editor Percepts User Reasoning Interface Editor Game Model Editor Task Editor Game Model Description Task Descriptions Perf. Task Evaluation Interface Evaluator Action / Control Translator (Mapper) Learning Outputs Actions Translated Model (Subset) Learning Task Game Interface Description Reasoning Interface Description Learning Translator (Mapper) Current State Model Updater Database Evaluation Methodology Description Stored State Advice Interface Database Engine State Controller Selected Reasoning System TIELT’s Internal Communication Modules

14. 14 Integrating Learning in Interactive Gaming Simulators Sensing the Game State (City placement example) TIELT Game Interface Editor Sensors User Game Model Editor Game Model Description Updates Game Interface Description Action Translator Actions Game Engine Game Engine Current State 1 2 4 3 4 In Game Engine, the game begins; a colony pod is created and placed. 1 The Game Engine sends a “See” sensor message identifying the pod’s location. This message template provides updates (instructions) to the Current State, telling it that there is a pod at the location See describes. 4 2 The Model Updater receives the sensor message and finds the corresponding message template in the Game Interface Description. 3 Controller Model Updater 3 The Model Updater notifies the Controller that the See action event has occurred. 5 5

15. 15 Integrating Learning in Interactive Gaming Simulators Getting Decisions from the Learning System (City placement example) TIELT Selected Reasoning System Selected Reasoning System Learning Module #1 Learning Module #n... User Learning Interface Editor Agent Editor Task Descriptions Learning Translator Translated Model (Subset) Reasoning Interface Description Action Translator Learning Outputs The Controller notifies the Learning Translator that it has received a See message. The Learning Translator finds a city location task, which is triggered by the See message. It queries the controller for the learning mode, then creates a TestInput message to send to the reasoning system with information on the pod’s location and the map from the Current State. The Reasoning System transmits output to the Action Translator. The Learning Translator transmits the TestInput message to the Reasoning System. 1 22 3 4 Controller Current State 1 4 2 3

16. 16 Integrating Learning in Interactive Gaming Simulators TIELT Game Interface Editor User Action Translator Actions Game Engine Game Engine 1 2 4.a The Action Translator receives a TestOutput message from the Reasoning System. The Action Translator finds the TestOutput message template, determines it is associated with the city location task, and builds a MovePod operator (defined by the Current State) with the parameters of TestOutput. The Game Engine receives Move and updates the game to move the pod toward its destination, or The Action Translator determines that the Move Action from the Game Interface Description is triggered by the MovePod Operator and binds Move using information from MovePod. Reasoning Interface Editor 3 Game Interface Description Reasoning Interface Description Advice Interface The Advice Interface receives Move and displays advice to a human player on what to do next, or makes a Prediction. 4.b, c 1 4.a 2 3 Acting in the Game World (City placement example) 4.b Current State 2 Prediction Interface 4.c 3

17. 17 Integrating Learning in Interactive Gaming Simulators Initial Work Status (July 2004) TIELT v1 + documentation due 9/04 –Message protocols Current: Console I/O, TCP/IP Future: Library calls, HLA interface, RMI (possibly) –Message content: Configurable Instantiated templates tell it how to communicate with other modules –Initialization messages: Start, Stop, Load Scenario, Set Speed –Game Model representations (w/ Lehigh University) Simple programs TMK process models PDDL (language used in planning competitions) Stratagus/Wargus module (Lehigh University) Initial publicity (BRIMS’04, here) Workshops being planned: ICCBR’05 (plus competition), ICML’05,...? TIELT v1 + documentation due 9/04 –Message protocols Current: Console I/O, TCP/IP Future: Library calls, HLA interface, RMI (possibly) –Message content: Configurable Instantiated templates tell it how to communicate with other modules –Initialization messages: Start, Stop, Load Scenario, Set Speed –Game Model representations (w/ Lehigh University) Simple programs TMK process models PDDL (language used in planning competitions) Stratagus/Wargus module (Lehigh University) Initial publicity (BRIMS’04, here) Workshops being planned: ICCBR’05 (plus competition), ICML’05,...? Example

18. 18 Integrating Learning in Interactive Gaming Simulators TIELT Collaboration Projects (2004-05) OrganizationGame Interface and Model Reasoning Interface Tasks and Evaluation Methodology Mad Doc SoftwareEmpire Earth 2 (RTS) Troika GamesTemple of Elemental Evil (RPG) ISLESimCity (~RTS)ICARUSICARUS w/ FreeCiv, design Lehigh U.Stratagus/Wargus (RTS), and HTN/TMK designs Case-based planner (CBP) Wargus/CBP NWUFreeCiv (discrete strategy), and qualitative game representations U. MichiganSOARSOAR w/ 2 games (e.g., FSW, ToEE), design U. Minnesota-DuluthRoboCup (team sports)Advice-taking components Advice processing USC/ICTFull Spectrum Command (RTS) SOAR with FSC UT ArlingtonUrban Terror (FPS)DCA (lite version) UT AustinNeuroevolutione.g., Neuroevolution/EE2

19. 19 Integrating Learning in Interactive Gaming Simulators TIELT’s Internal Communication Modules TIELT’s KB Editors TIELT’s KB Editors Selected/Developed Knowledge Bases Game Model Description Task Descriptions Game Interface Description Reasoning Interface Description Evaluation Methodology Description TIELT’s User Interface Prediction Interface Evaluation Interface Coordination Interface Advice Interface TIELT User TIELT User TIELT Collaborations (2004-05) Knowledge Base Libraries Game Library Mad Doc EE2 ToEE Troika FreeCiv NWU ISLE Platform Library Stratagus Lehigh U. UT Arl. FSC/R USC/ICT Urban Terror U. Minn-D. RoboCup Reasoning System Library Learning Modules Soar: U.Mich ICARUS: ISLE DCA: UT Arlington Neuroevolution: UT Austin Others: Many LU, USCMich/ISLEU. Mich.Many U.Minn-D.USC/ICT U.Mich.

20. 20 Integrating Learning in Interactive Gaming Simulators Summary: Questions? Concerns? TIELT: Mediates between a (gaming) simulator and a learning-embedded reasoning system Goals: –Simplify running learning expts with cognitive systems –Support DARPA challenge problems in learning Designed to work with many types of simulators & reasoning systems TIELT: Mediates between a (gaming) simulator and a learning-embedded reasoning system Goals: –Simplify running learning expts with cognitive systems –Support DARPA challenge problems in learning Designed to work with many types of simulators & reasoning systems Status: v1 scheduled for completion in 9/04 –Please see Matt Molineaux’s demo 11 additional organizations about to start 1-year collaborations –Enhances probability that TIELT will achieve its goals Status: v1 scheduled for completion in 9/04 –Please see Matt Molineaux’s demo 11 additional organizations about to start 1-year collaborations –Enhances probability that TIELT will achieve its goals