Institutionalizing a Culture of Statistical Thinking in DoD Test & Evaluation: The Role of the Statistician in Leading Change Dr. Laura Freeman

Innovation Adoption Dr. Eric Schmidt, Testimony to House Armed Services Committee April 17, 2018

The DoD is pushing for more innovation Big Data Machine Learning Artificial Intelligence Cybersecurity Autonomy …and many more

Statistician’s are uniquely equipped to lead & implement change, especially in data-centric fields! Laura’s conjecture

Kotter’s Process for Leading Change Establish a sense of urgency Form a powerful coalition Create a vision Communicate the vision Empower others to act Create short term wins Consolidate improvements and produce more change Institutionalize new approaches

Improving Operational Testing: A case study from my past 8 years

Goal of Operational Test: Evaluate Operational Effectiveness, Suitability, and Survivability Operational Environment Representative Users “Real” Threats Conducting Missions

Why did we need to improve test methods? Figure from DOT&E EA-18G BLRIP Figure from DOT&E EA-18G BLRIP Percent Success

But this was not a new idea! 1998 National Research Council Study Process Recommendations: OT&E input on requirements Sequential testing (multiple small scale tests) Data storage Technical Recommendations: Test planning (use “state-of the- art experimental design”) Estimates of uncertainty Use all relevant information Appropriate statistical models

Why were we successful this time? Establish a sense of urgency – approval to conduct testing required the use of experimental design

Laying the foundations for statistical methods in T&E 2.Form a powerful coalition Research Consortium Offsite Meeting Charter Statistical Engineering with NASA

3. Create a vision

4. Communicate the vision – case studies & training F-35 Close Air Support DOE

5. Empower others to act & create short term wins Bonus - Sharing lessons learned advanced our mutual understanding of the vision xc v xc

7. Consolidate & Produce More Change Goals/Accomplishments: Assess the OTA workforce size, capabilities, education and new hire needs Roadmap for training/education and other OSD support needed to increase test design and analytic capabilities Case Studies! Examples of statistical design and analysis techniques appropriate for T&E Guidance for the documentation of test design and statistical rigor in TEMPs, Test Plans and Reports. TEMP Guidebook DOE memos Developed Best Practices for good test design and analysis of test data Formation of advisory board to support DT and OT communities

We continue to increase the statistical defensibility of DoD Test and Evaluation 2017 2016 National Research Council Study Design of Experiments endorsed as a sound methodology for OT&E OTA MOA on DOE DOT&E Initiatives Guidance on DOE in TEMPs DOT&E Policy Issued OTA Test Design Processes Updated DOT&E Science Advisor Established “Test Science Roadmap” effort DOT&E/ TRMC funded Science of Test Research Consortium DOT&E TEMP Guide Published DASD (DT&E) STAT Implementation Plan STAT COE DOT&E Roadmap Report Two Additional DOT&E Guidance memos on Application of DOE to OT&E Survey Best Practices Memo Cyber- security Procedures Additional Survey and cyber work Modeling and simulation validation guidance Cyber priorities Updated TEMP Guidance M&S Guidance 2015 2009 2010 2011 2012 2013 1998 2014

Shameless Plug: Testscience.org & DATAWorks

Laura’s conjecture revisited Statistician’s are uniquely equipped to lead & implement change, especially in data-centric fields… But to solve the high impact problems, we need more. Statistical Engineering – the study of systematically integrating statistical concepts, methods, and tools with other relevant disciplines to solve important problems sustainably.

Thank you!

Statistical Thinking Emphasis on the process not the tools! Statistical thinking is a philosophy of learning and action based on the following principles: All work occurs in a system of interconnected processes Variation exists in all processes Understanding [characterizing] and reducing variation are keys to success Emphasis on the process not the tools!

Statistical Thinking for “Big Data”

We take a statistical approach to operational testing Carrier variant Conventional Short takeoff/vertical landing

We take a scientific approach to operational testing Carrier variant Conventional Short takeoff/vertical landing Mission Areas Air Threat Ground Threat Air-Surface Strike Destruction/Suppression of Enemy Air Defenses Defensive counter air Offensive counter air Close air support Search and rescue JSF OT designed around eight mission areas focusing on the capability the F-35 provides Operational space = 8 separate mission areas Separate test design within each mission area Threat continuum across mission areas Comparison test for A/S attack, D-SEAD, CAS, CSAR, and FAC-A

We take a scientific approach to operational testing Carrier variant Conventional Short takeoff/vertical landing Power estimates Large detectable difference Medium detectable difference Small detectable difference Number of tests 30 1

Weapons Production Facility Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense Weapons Production Facility Note that F-35 is a 5th gen platform. Old notions of sole Strike, OCA, and D-SEAD separately don’t make sense. A 5th gen multi-mission platform can serve as both the strike package and the escort package. But then, how do we cover such a vast complex mission space efficiently? – well we can execute the missions together as one approach – here is what that would look like. So we want to take out a weapons production facility – it is most likely protected by red air (see two)

Weapons Production Facility Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense Radar Radar Surface to Air Missile Weapons Production Facility But not only red air, they also have an Integrated air defense… Radar Surface to Air Missile Radar

Weapons Production Facility Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense Radar Radar Surface to Air Missile Weapons Production Facility Those surface to air missiles protect and prevent access to the weapons facility Radar Surface to Air Missile Radar

Weapons Production Facility Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense Radar Radar Surface to Air Missile Weapons Production Facility But then the F-35’s show up (one 4-ship to take out the SAM), and other to drop bombs on the weapons facility Radar Surface to Air Missile Radar

Weapons Production Facility Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense Radar Radar Surface to Air Missile Weapons Production Facility Look the SAM ring got smaller, we suppressed it!!! Radar Surface to Air Missile Radar

Weapons Production Facility Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense Radar Radar Surface to Air Missile Weapons Production Facility Look the SAM ring got smaller, we suppressed it!!! Radar Surface to Air Missile Radar

Weapons Production Facility Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense Radar Radar Surface to Air Missile Weapons Production Facility Note that F-35 is a 5th gen platform. Old notions of sole Strike, OCA, and D-SEAD separately don’t make sense. A 5th gen multi-mission platform can serve as both the strike package and the escort package. But then, how do we cover such a vast complex mission space efficiently? – well we can execute the missions together as one approach – here is what that would look like. So we want to take out a weapons production facility – it is most likely protected by red air (see two) But not only red air, they also have an Integrated air defense… Those surface to air missiles protect and prevent access to the weapons facility Look the SAM ring got smaller, we suppressed it!!! But red air responds, not there is some mix of air-to-air, which seemed to hard to illustrate and the strike package proceeds to the target. Radar

Characterization across operational envelope – Response Variables Targeting Accuracy Striker Striker First Track Range Striker First Hostile Declaration Range Striker First Shot Range Red Air First Detection Range Red Air First Shot Range Striker SAM Track Time Proportion of Valid Weapon Releases to Number of Valid Weapon Releases Required to Meet Mission Tasking Proportion of Assigned Air to Surface Targets Removed Proportion of Striker Kill Removed Striker to Red Air Exchange Ratio Geolocation Find Time Fix Time DEAD Time Targeting Accuracy Escort Escort SAM Track Time Proportion of Assigned SAM Elements Removed Proportion of Assigned SAM Elements Engaged Exchange Ratio Closest Red Air Range to Strike Package Blue Striker Encroachment Range Escort First Track Range Escort First Hostile Declaration Range Escort First Shot Range Proportion of Escort Blue Strikers that reach their Weapons Release Point Proportion of Protected Aircraft (Strikers) Not Kill Removed Proportion of Escort F-35 Kill Removed Escort to Red Fighter Exchange Ratio Lots of measures to capture: Mission outcomes Air to Air Performance Air to Surface Performance System sensor capabilities The point is not the variables themselves, but that we need lots of them! Most tests do!

Experimental designs determine test adequacy 24 Run, D-Optimal 2nd Order Design Disallowed Combinations

Two mission designs, executed in a 5th generation scenario Critical points – coverage of the space is determined by test team experience and necessary interactions. Both test designs have 24 runs, matched together in two-four ship executions. Power and interactions were critical for deciding on 24.