1. Institutionalizing a Culture of Statistical Thinking in DoD Test & Evaluation: The Role of the Statistician in Leading Change
Dr. Laura Freeman

2. Innovation Adoption
Dr. Eric Schmidt, Testimony to House Armed Services Committee April 17, 2018

3. The DoD is pushing for more innovation
Big Data Machine Learning Artificial Intelligence Cybersecurity Autonomy …and many more

4. Statistician’s are uniquely equipped to lead & implement change, especially in data-centric fields!
Laura’s conjecture

5. Kotter’s Process for Leading Change
Establish a sense of urgency
Form a powerful coalition
Create a vision
Communicate the vision
Empower others to act
Create short term wins
Consolidate improvements and produce more change
Institutionalize new approaches

6. Improving Operational Testing: A case study from my past 8 years

7. Goal of Operational Test: Evaluate Operational Effectiveness, Suitability, and Survivability
Operational Environment Representative Users “Real” Threats Conducting Missions

8. Why did we need to improve test methods?
Figure from DOT&E EA-18G BLRIP
Figure from DOT&E EA-18G BLRIP
Percent Success

9. But this was not a new idea! 1998 National Research Council Study
Process Recommendations:
OT&E input on requirements
Sequential testing (multiple small scale tests)
Data storage
Technical Recommendations:
Test planning (use “state-of the- art experimental design”)
Estimates of uncertainty
Use all relevant information
Appropriate statistical models

10. Why were we successful this time?
Establish a sense of urgency – approval to conduct testing required the use of experimental design

11. Laying the foundations for statistical methods in T&E
2.Form a powerful coalition
Research Consortium
Offsite Meeting
Charter
Statistical Engineering with NASA

12. 3. Create a vision

13. 4. Communicate the vision – case studies & training
F-35 Close Air Support DOE

14. 5. Empower others to act & create short term wins Bonus - Sharing lessons learned advanced our mutual understanding of the vision xc v xc

15. 7. Consolidate & Produce More Change
Goals/Accomplishments:
Assess the OTA workforce size, capabilities, education and new hire needs
Roadmap for training/education and other OSD support needed to increase test design and analytic capabilities
Case Studies!
Examples of statistical design and analysis techniques appropriate for T&E
Guidance for the documentation of test design and statistical rigor in TEMPs, Test Plans and Reports.
TEMP Guidebook
DOE memos
Developed Best Practices for good test design and analysis of test data
Formation of advisory board to support DT and OT communities

16. We continue to increase the statistical defensibility of DoD Test and Evaluation
2017
2016
National Research Council Study
Design of Experiments endorsed as a sound methodology for OT&E
OTA MOA on DOE
DOT&E Initiatives
Guidance on DOE in TEMPs
DOT&E Policy Issued
OTA Test Design Processes Updated
DOT&E Science Advisor Established “Test Science Roadmap” effort
DOT&E/ TRMC funded Science of Test Research Consortium
DOT&E TEMP Guide Published DASD (DT&E) STAT Implementation Plan
STAT COE
DOT&E Roadmap Report
Two Additional DOT&E Guidance memos on Application of DOE to OT&E
Survey Best Practices Memo
Cyber- security Procedures
Additional Survey and cyber work
Modeling and simulation validation guidance
Cyber priorities
Updated TEMP Guidance
M&S Guidance
2015
2009
2010
2011
2012
2013
1998
2014

17. Shameless Plug: Testscience.org & DATAWorks

18. Laura’s conjecture revisited
Statistician’s are uniquely equipped to lead & implement change, especially in data-centric fields… But to solve the high impact problems, we need more. Statistical Engineering – the study of systematically integrating statistical concepts, methods, and tools with other relevant disciplines to solve important problems sustainably.

19. Thank you!

21. Statistical Thinking Emphasis on the process not the tools!
Statistical thinking is a philosophy of learning and action based on the following principles:
All work occurs in a system of interconnected processes
Variation exists in all processes
Understanding [characterizing] and reducing variation are keys to success
Emphasis on the process not the tools!

22. Statistical Thinking for “Big Data”

23. We take a statistical approach to operational testing
Carrier variant
Conventional
Short takeoff/vertical landing

24. We take a scientific approach to operational testing
Carrier variant
Conventional
Short takeoff/vertical landing
Mission Areas
Air Threat
Ground Threat
Air-Surface
Strike
Destruction/Suppression of Enemy Air Defenses
Defensive counter air
Offensive counter air
Close air support
Search and rescue
JSF OT designed around eight mission areas focusing on the capability the F-35 provides
Operational space = 8 separate mission areas
Separate test design within each mission area
Threat continuum across mission areas
Comparison test for A/S attack, D-SEAD, CAS, CSAR, and FAC-A

25. We take a scientific approach to operational testing
Carrier variant
Conventional
Short takeoff/vertical landing
Power estimates
Large detectable difference
Medium detectable difference
Small detectable difference
Number of tests 30 1

26. Weapons Production Facility
Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense
Weapons Production Facility
Note that F-35 is a 5th gen platform. Old notions of sole Strike, OCA, and D-SEAD separately don’t make sense. A 5th gen multi-mission platform can serve as both the strike package and the escort package. But then, how do we cover such a vast complex mission space efficiently? – well we can execute the missions together as one approach – here is what that would look like.
So we want to take out a weapons production facility – it is most likely protected by red air (see two)

27. Weapons Production Facility
Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense
Radar
Radar
Surface to Air Missile
Weapons Production Facility
But not only red air, they also have an Integrated air defense…
Radar
Surface to Air Missile
Radar

28. Weapons Production Facility
Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense
Radar
Radar
Surface to Air Missile
Weapons Production Facility
Those surface to air missiles protect and prevent access to the weapons facility
Radar
Surface to Air Missile
Radar

29. Weapons Production Facility
Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense
Radar
Radar
Surface to Air Missile
Weapons Production Facility
But then the F-35’s show up (one 4-ship to take out the SAM), and other to drop bombs on the weapons facility
Radar
Surface to Air Missile
Radar

30. Weapons Production Facility
Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense
Radar
Radar
Surface to Air Missile
Weapons Production Facility
Look the SAM ring got smaller, we suppressed it!!!
Radar
Surface to Air Missile
Radar

31. Weapons Production Facility
Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense
Radar
Radar
Surface to Air Missile
Weapons Production Facility
Look the SAM ring got smaller, we suppressed it!!!
Radar
Surface to Air Missile
Radar

32. Weapons Production Facility
Characterization across operational envelope – Strike, Offensive Counter Air, and Destruction/Suppression Enemy Air Defense
Radar
Radar
Surface to Air Missile
Weapons Production Facility
Note that F-35 is a 5th gen platform. Old notions of sole Strike, OCA, and D-SEAD separately don’t make sense. A 5th gen multi-mission platform can serve as both the strike package and the escort package. But then, how do we cover such a vast complex mission space efficiently? – well we can execute the missions together as one approach – here is what that would look like.
So we want to take out a weapons production facility – it is most likely protected by red air (see two)
But not only red air, they also have an Integrated air defense…
Those surface to air missiles protect and prevent access to the weapons facility
Look the SAM ring got smaller, we suppressed it!!!
But red air responds, not there is some mix of air-to-air, which seemed to hard to illustrate and the strike package proceeds to the target.
Radar

33. Characterization across operational envelope – Response Variables
Targeting Accuracy Striker
Striker First Track Range
Striker First Hostile Declaration Range
Striker First Shot Range
Red Air First Detection Range
Red Air First Shot Range
Striker SAM Track Time
Proportion of Valid Weapon Releases to Number of Valid Weapon Releases Required to Meet Mission Tasking
Proportion of Assigned Air to Surface Targets Removed
Proportion of Striker Kill Removed
Striker to Red Air Exchange Ratio
Geolocation Find Time
Fix Time
DEAD Time
Targeting Accuracy Escort
Escort SAM Track Time
Proportion of Assigned SAM Elements Removed
Proportion of Assigned SAM Elements Engaged
Exchange Ratio
Closest Red Air Range to Strike Package
Blue Striker Encroachment Range
Escort First Track Range
Escort First Hostile Declaration Range
Escort First Shot Range
Proportion of Escort Blue Strikers that reach their Weapons Release Point
Proportion of Protected Aircraft (Strikers) Not Kill Removed
Proportion of Escort F-35 Kill Removed
Escort to Red Fighter Exchange Ratio
Lots of measures to capture: Mission outcomes Air to Air Performance Air to Surface Performance System sensor capabilities
The point is not the variables themselves, but that we need lots of them! Most tests do!

34. Experimental designs determine test adequacy
24 Run, D-Optimal 2nd Order Design Disallowed Combinations

35. Two mission designs, executed in a 5th generation scenario
Critical points – coverage of the space is determined by test team experience and necessary interactions. Both test designs have 24 runs, matched together in two-four ship executions.
Power and interactions were critical for deciding on 24.