Modelling CGFs for tactical air-to-air combat training Motivation-based behaviour and Machine Learning in a common architecture Jan Joris Roessingh, Ph.D., Maj. Roel Rijken, M.Sc. National Aerospace Laboratory (NLR), Royal Netherlands Air Force smartbandits@nlr.nl

Contents Part 1(Intelligent CGFs) Smart Bandits Requirements State-of-the-art in CGFs Architecture Cognitive models Part 2 (Machine Learning) Experiment RL (example) Pros & Cons ML Towards hybrid models (example) Conclusions Exploratory Team under IST panel

Project Goals “Smart Bandits” Development of intelligent Computer Generated Forces for tactical mission training of fighter pilots in the opponent role ‘humanlike’ behaviour capable of tactical reasoning intelligent decision making team work .. Constrained by Situation Awareness Memory capacity Intelligent CGFs should be suitable for use in simulations at MoD

Application of intelligent CGFs in Embedded Training?

Current Scope Tactical mission training Tactics are techniques for using aircraft and weapons in a combined fashion with the purpose to gain advantage over / defeat the enemy Air-to-Air Beyond Visual Range (>10 NM) Offensive and Defensive Counter Air (picture) 1v1, 2v2, 4v4 engagements

Requirements Operational CGFs in the opponent role should be autonomous should exhibit credible behaviour should contribute to training value of simulation Functional weapon system functions human functions more specific functions per mission phase planning & briefing targeting executing the game plan self-defence

Research facility: NLR’s “Fighter 4-Ship” One station of the Fighter 4-Ship (Four networked F-16 simulations)

F-16 executes OCA mission (Offensive Counter Air) Su-27 executes DCA mission (Defensive Counter Air) FLOT

State-Of-the-Art in CGFs Scenario-management packages behaviour of CGFs is “scripted” pre-defined CGFs lack appropriate weapon and human models limited possibilities for the use of AI Agent Qualities Non-responsive behaviour Stimulus-Response (S-R) behaviour Delayed Response (DR) behaviour motivation-based behaviour combines S-R and DR behaviour + ‘motivational states’ TAC-AIR SOAR cognitive architecture deals with observations, decisions and coordination Order of magnitude: 10.000 tactical decision rules Machine Learning techniques

Agent Development Approach Cognitive (BDI) Models Machine Learning Techniques Tactical Scenarios (scripted) Single Agent - 1v1 Situation Awareness Evolutionary techniques Theory of Mind Reinforcement Learning Multi Agent - 2v2 Decision Making Neural Networks Multi Agent - 4v4 Smart Adversary Behaviour

Architecture Simulator – CGF package Agent-models are functionally separated from simulation environment Human-like behaviour can be linked to CGFs Different agent models can run on different machines

Cognitive Models (Team) Situation Awareness Naturalistic Decision Making Theory of Mind

Example : Situation Awareness Definition Mica Endsley (1988) three levels of SA: the perception of the environment, the comprehension and integration of information, and the projection of information into future events. Translation to “BDI” framework Perceive: Observations/ Simple beliefs Understand: Complex beliefs Anticipate: Future beliefs Human constraints belief formation constrained by workload

Cognitive model for situation awareness: overview from Hoogendoorn, van Lambalgen & Treur, 2011

Example belief network for SA model from Hoogendoorn, van Lambalgen & Treur, 2011

Reinforcement Learning Experiment

Pros and Cons Machine Learning Save development time (less knowledge elicitation required) Adaptation to environment and opponent Complex behaviour in complex domains New tactics and evaluation of human tactics Cons Learning speed Effectiveness (unpredictable behaviour) Computation time and memory requirements Adaptation to game randomness Increase development time (tweaking)

Hybrid models (Dynamic Scripting, Spronck et al., 2005) Reinforcement learning Scripts ‘Adaptive game AI with dynamic scripting’, Spronck et al., 2005

Conclusions Cognitive modelling one of the fundamental techniques for motivation-based behaviour CGFs Machine Learning is powerful tool to: enhance and complement cognitive models reduce knowledge elicitation efforts Smart Bandits: combination of models, utilizing advantages of different approaches

Let us know whether you are interested to participate! Technical Activity Proposal (TAP) Machine Learning Techniques for Battlefield Agents Exploratory Team under the IST panel Some topics to be covered: Current applications of ML Potential applications in Defence (all Forces) Potential barriers for application Most appropriate ML techniques Systems engineering aspects of ML 3 meetings in 2012, 1st in Amsterdam, early 2012 Leading to a Research Task Group TAP available! Let us know whether you are interested to participate!