1. Ascertaining Validity in the Abstract Realm of Constructive Simulation Models: An Analysis of the MAGTF Tactical Warfare Simulation (MTWS) Major Donald Herod, USMC In Partnership with SEED Advisor: William Becker, Ph.d Co-Advisor: Prof Curtis Blais

2. As a simulations community, we do a very good job of verifying and validating algorithmic functionality and “real world” platform-based performance in a constructive simulation As a simulation community, we have NEVER determined how to conduct a holistic validation of a constructive simulation so as to determine if the model as a training tool produces “real world” responses/outputs. This can lead to negative training transfer and unnecessary loss of life in combat. The Problem

3. The Marine Corps premier constructive simulation… capable of full range of MAGTF combat, combat support and combat service support applications, including amphibious operations. MTWS… still meeting the doctrinal expectations of the current operational environment.

4. System Overview Constructive, stochastic, aggregate level simulation of ground, air, maritime, and amphibious operations Computer-assisted command and control training system Provides exercise control services and tactical combat simulation –Multi-sided, real-time –Full range of battle space capabilities –All phases of military operations

5. System Benefits User-definable parametric data (characteristics and behaviors) Global play box; user-definable regions for digital terrain data Tool for course of action evaluation in operational planning

6. Thesis Overview Research Question: Can the outputs provided by a constructive simulation be data-farmed to determine if the model’s performance parameters produce valid outputs that stimulate the training audience? Additionally, is there a range of values or specific design points that cause the system to produce algorithmic anomalies (verify)? Abstract: The requirement to routinely assess staff training simulations is necessary to ensure the interaction within the simulation provides a realistic training environment. This project will explore methods to apply data farming to MTWS to obtain information about its computational characteristics as part of a more general study into use of data farming for simulation validation.”

7. Getting Started Scenario Development –Think small… simulation is like Pandora's Box –What is the training objective? Identify Factors of Interest –Deep dive into the parametrics that govern computations in the model –The number of factors will determine the number of Design Points required. Methodology –Design of Experiments (DOE) Nearly Orthogonal Latin Hyper-cubes (NOLH) –Batch File Replication –Simulation Run (AKA: Micro-Battles) –Data Mining –Statistical Analysis/Prediction Profiling

8. 8 Orthogonal Latin Hyper-cube

9. MTWS Output 9

10. Multivariate Correlation & Scatterplot 10

11. 3D Scatterplot of the Sample Space 11

12. Stepwise Fit Model 12

13. Stepwise Fit Model 13

14. Prediction Profiler 14

15. Casting a wider net… –Data Farm all User Adjustable Factors of Interest NOLH with 20 factors of interest (10 per side) Future Work - K-Kill Hardness - Mounted Hardness - F-Kill Hardness - Cbt Power - Hit Probability - Firing Rate - M-Kill Hardness - Vulnerability - Destructive Pwr - Range DOE containing 129 DP (30 reps per DP) 3870 Micro-Battles –Generate MTWS Batch Files Trouble-shoot errors and dependencies –Stepwise Fit a Model with interactions Generate Prediction Profiles for Factors on Interest

16. Impacts Present a new approach to VV&A that provides a holistic assessment of stochastic constructive simulations. Recommend enhancements to MTWS based on observed anomalies to verify that the model is producing expected outcomes. Generate a more scientific means of determining appropriate parametircs that are valid for a specific training requirement.

17. Questions