Joost Westra, Frank Dignum,Virginia Dignum Scalable Adaptive Serious Games using Agent Organizations

Overview  Introduction  Adaptation to the trainee  Organized adaptation of agents  Scalability  Conclusions

Serious Gaming

Dynamic Difficulty Adjustment  Online adaptation: Continuously balance challenges in the game with (developing) skills of the trainee

Current approaches  Fixed difficulties  Central control or no coordination  Mainly adjust simple subtasks

Agent Approach  Agents part of the design process  Reasoning agents  Adapting agents  Specify boundaries of the adaptation (agent organization) Example: Trainee is fire commander 2 fireman agents 1 victim agent 1 agent controlling spreading of fires

Aspects  User Evolving skills (when learning)  Characters Characters adapt independently Characters active for long periods, so, adaptation should be believable  Keep storyline Learning goals have to be maintained!  Adaptation must be coordinated!  Performance can not be measured separately for each skill and influence of each agent

Story-line  Guarantee certain states are reached  Subtasks defined by scene scripts and landmarks  Connected by interaction structure Describes game progress Connecting scenes Tasks in parallel Start Get Access to Room Evacuate Victim Extinguish Fire End

Adaptation Engine  Coordinates task difficulty  Check with game model  Combinatorial auction User model Agent preferences Agent model 2APL Agent Agent model Agent Bidding User Model Adaptation Engine Update Plans Bid Task WeightsSkill Levels Selection Preferences & Temination Scene States Applicable plans Game Model StartGet to site Gather info Secure area Search building Evacuate victims Extinguish fire Clear areaEnd

Agent Perspective  Agents Propose actions to adaptation engine at “natural” synchronization points  Created to facilitate trainee’s objectives (optimize agent behavior relative to trainee’s performance!)  Not responsible for suitable combination  Conflict: Stay as consistent as possible Propose enough actions  Adaptation engine can request agents to terminate behavior if necessary for coordination

Framework Agent model 2APL Agent Agent model Agent Bidding Game world Game state Agent interface User Model Adaptation Engine NPC Update User Performance Translate Plans Bid Update Beliefbase Task WeightsSkill Levels External Action SelectionGame Actions Preferences & Temination Scene States Applicable plans Game Model

Scalabilty: Scenes  Agents can only execute plans of active scenes  Partial ordering gives a relatively low number of concurrent scenes  Sub-scenes: Even more fine grained pre-selection Gather Info Search Building Secure Area Evacuate Victims Extinguish Fire Get to site Multiple victims EndStart Kitchen Fire

Scalabilty: Agent implementation  Active Subscenes are put in beliefbase  Only plans with active sub-scenes are applicable  Only plans for current goals are applicable  Other active beliefs can also restrict the number of applicable plans

Scalabilty: Believability  Agents estimate the Believabilty for each applicable action  Some actions clearly ruin the Believabilty of the agents Believabilty 0 -> never suggest Higher threshold than 0 is usually advisable -> even lower number of suggestions  Influence becomes bigger if the game progresses The player has more expectations on the behavior of the NPC

Scalabilty: Combination boundaries  Game model requirements can decrease the number of checked combinations  Influence greatly depend on the restriction Easy: – At least one fireman should perform X – Only one fireman available – Only evaluate combinations with the fireman performing X Difficult – X needs to be performed by at least two agents – No real indicator that other plans might not be better.

Scalability: Example  Assumptions 30 different scenes – 2 active at the same time 4 subscenes – 2 active at the same time 6 plans per subscene per agent  Results Naive: – 720 active plans(30 scenes*4 sub-scenes*6 actions per sub-scene) Agent Organization: – 12 active plans (6 actions per sub-scene *2 sub-scenes active per scene * 2 active scenes /2 for believability filtering) times as fast with only four agents

Scalability: Example, number of Agents

Conclusion  Continuous adaptation to the trainee  Agent based approach Complex individual behavior and adaptation possible  Agent organization for coordination Balance between individual flexibility and global story line maintaining learning goals Minimal central control for more efficiency and more flexibility  More scalable than centralized approach

Questions? Thank You!

Agent Implementation  Adaptable BDI-agents (2APL extension) Equivalent plans Preference relation Environmental information also used Search building Evacuate victims Pairwise search Parallel search Thorough serch One by one Groupwise Front/back rooms floors Front/back parallel Rooms parallel Floors parallel Top floor first Utility rooms first Living quarter first

Scene specification  Roles of the agent  Ordering of tasks  Requirements on task Start Get Access to Room Evacuate Victim Extinguish Fire End

Task Difficulty  Task dependent on behavior of the agents  Behavior variations are fixed  Estimated by domain expert  Updated by offline learning  Much faster adaptation 0,30,50,7

Sample code

Example: User model  Keeps track of the user skill levels  Just updated Out of scope  Three skills: Extinguishing fire Giving orders to his team and Extract victims.

Example: Different branches (2/2)  Not only different agent behavior: Different environment Different NPC’s …... Team & Fire : Team focus … Team & Fire : Balanced Team & Fire : Fire focus Evaluation Decision Priority 1teamOrders < 0.5 2fireFire <0.3 3balanced-

Framework Agent model 2APL Agent Agent model Agent Bidding Game world Game state Agent interface User Model Adaptation Engine NPC Update User Performance Translate Plans Bid Update Beliefbase Task WeightsSkill Levels External Action SelectionGame Actions Preferences & Temination Scene States Applicable plans Game Model

Example: Different branches (1/2)  Focus more on certain skills  Different transition states Skill level Game state  Still adaptation within different scenes …... Team & Fire : Team focus … Team & Fire : Balanced Team & Fire : Fire focus Evaluation Decision

Example: Adaptation Coordination  Combined difficulty of tasks Domain dependent  Send bids, check model, find best fit Could be asynchronous  Skill levels, agents preferences, game model  Multiple fires are more difficult than one  Still possible to have different fire types at the same time depending on the skill level

Example: Task Coordination  Agent prefences and skill level combination do not always fit game model  Each carry one side of stretcher  Agents send bid of their preferred action Both prefer to operate hose One capable op stabilizing victim Carry Stretcher Operate Hose End Stabilize Victim

Adapt the game to the user Beginner: FrankExpert: Joost A user only learns if performing on his own level