Scott Randall Thompson, MSCS, MBA, PMP srthompson@alionscience.com EXTENDED ANALYTICS FOR JTLS-GO® Scott Randall Thompson, MSCS, MBA, PMP srthompson@alionscience.com 12 December 2018

Agenda M&S Outcome Overview JTLS-GO Enhancements to Support Outcome Generation Alion Outcome Analysis Overview and Development Roadmap Objectives: Provide high-level, minimal-technical overview of outcome data analysis initiatives Additional information available upon request outside of briefing

High-Level Overview M&S process, generally speaking… M&S tools take in a ‘model’ and produce an ‘outcome’ Outcomes are result table(s) in database, usually, from a run For live or virtual events, M&S tools also generate outputs which can then be analyzed using pre-built tools or visualizations Extracting analytic results is often a manual process Alion is working on ‘automating’ and providing visualization of analytic results

Randomness (Abbreviated) In case you ever wondered why/how ‘same run’ gets ‘different results’ Random number generators! (Well, usually Pseudo Random Numbers…) Variables that resolve per-run to a number in a distribution JTLS-GO handling: Attributes of the model have a statistical distribution and range for certain elements The JTLS design team has several Operations Research trained people that decide what is the best distribution to represent the stochastic process being implemented.  Examples! When computing the direction of an impact point of a weapon to its aimpoint, a uniformly distributed random variate between 0.0 and 360.0 is  drawn. Note a second random variate is drawn using a Normal Distribution based on the Circular Error Probable (CEP) database parameter for the type of weapon or firing system. Naturally 50% of the weapons impacts will fall within this CEP and 50% outside of the CEP, as per the definition of CEP. The number of weapon hits needed to sink a ship draws from a Weibull distribution. JTLS started at the Naval Postgraduate School and the professors there insisted that the Weibull distribution be used because they were convinced that it best represented the stochastic event of sinking a ship. This means that the database builder must provide the Weibell Shape and Weibull Scale parameter for each Ship Unit Prototype.

JTLS-GO Usage Overview JTLS-GO Tool Usage “Today” (from JTLS-GO materials) Many organizations perform OPLAN Evaluation Identical operational scenarios may be repeated many times with a few minor ‘model’ changes in order to evaluate alternative outcomes Analytic use of JTLS demands that the characteristics and behaviors of all modeled capabilities accurately reflect real world parameters. The results need to be accurate to allow meaningful comparison. Alion Understanding of JTLS-GO Usage Most/All JTLS-GO runs are human-in-the-loop Data is saved at each checkpoint and end-state made available in the AAR tables / tool JTLS-GO runs do not occur in parallel (random seeding, saved tables, etc.)

Automated Post-Processing Alion’s Concept Current State Why did I receive this M&S result from this single run? Why did my plan succeed in meeting objectives? Why did the results fail to meet objectives? M&S Outcome Insights From Built-in Analysis Analyst Manual Post- Processing Analyst Future State I understand cause of this M&S result I know why my plan succeeded in meeting objectives I know why the results failed to meet objectives I know what modifications to make to progress toward meeting objectives M&S Outcomes Analyst New Insights From Analysis Automated Post-Processing

Benefits of These Capabilities Higher-Level Benefits: Scenario validation (“IVV”) Ability to establish a scenario baseline Ability to pre-vet scenarios before live/virtual events Specifics: Automate standard statistical reporting so that analysts can parse data faster. Automation of a standard set of analysis use cases of interest to most/all M&S scenarios Use of advanced techniques to determine causal factors Additional visualization (map, histograms, bar charts, selecting additional reports to be generated, analytic dashboard) (And then scenario specific analysis can also be developed) (And other use cases that may be of interest… outliers, etc.)

JTLS-GO Tool Enhancements JTLS-GO version 5.0 supports a Live/Virtual run with orders Has to run ‘serially’ due to random number seeding Cannot launch multiple runs automatically This means generating N=30* number of runs will take a really long time Under ideal conditions, 1 run of given scenario = ~12-14 hours or 3 game days** Taking about 15-16 days to conduct 30 runs to achieve statistically significant outcome Alion’s JTLS-GO analytic enhancements in 5.1***: Changing seed on multiple runs at launch to run in parallel; 3 runs at a time Throughput now means N=30 can be achieved in an estimated 5 days instead of 15 AAR data will be saved for each run, then the process will continue to the next run Enhanced capability provides data for all 30 runs *N=30 rule confirmed by T distribution **Korean peninsula ‘demo’ scenario provided ***Final release number pending

JTLS-GO Analyzing Data Outcomes Alion Approach Ingest outcomes from M&S tools Process and apply use-cases on them Attempt to provide ‘generic’ use cases and reporting of value Provide scenario-specific use cases as needed Generate reports N=30 runs provides enough data to analyze some statistical items for several use cases BLUE objectives achieved, RED objectives achieved Can do more if needed (outliers, more complicated use cases) Analytic Capabilities Automated standard statistical reporting Advanced techniques to determine causal factors Analytic dashboard to assist analysts

First Full Demonstration Development Roadmap Dec 2018 Alpha Version Apr 2019 Beta Version Jul 2019 Version 1.0 First Full Demonstration Feb 2020 Version 1.1 Fully Operational (Note - only using Korean Peninsula Scenario) Dec 2018 - Automated process, basic statistical reporting July 2019 – 1.0 GUI Available (development GUI available now)

Next Major Steps Capability Development to IOC/FOC Scenario Ingestion Partial/Full-Automation Additional Use Case Development Transition to Classified Environment to Run Classified Models

QUESTIONS? srthompson@alionscience.com