1. Collaboration to Meet Future T&E Needs ITEA 14 September 2010 1 Mr. Mike Crisp Deputy Director, Air Warfare Operational Test and Evaluation

2. Need vs. Speed vs. Cost: A Balancing Act in Acquisition Requirements Process stressed in Meeting Today’s Needs DoD Budgets Under Pressure Warfighter Demand Cycle inside Acquisition OODA Loop Concurrent Development, Production, Operations, and Support –Program Managers doing it all Configuration Control –How many one or two of a kind can we manage? DoD Acquisition Life Cycle Framework –Complex Systems Integration in Fluid Operational Environments –Adaptive and Flexible enough to support rapid fielding? JUONS, Rapid Block upgrades, Milestones, Increments Innovative and Adaptive Approaches Required

3. 3 Challenges for Building Effective Integrated Test Partnership Limited Test Resources: Range Capabilities, Size, Forces (Manpower & Materiel) Limited Test Time: Fielding & Production Schedules, Range Availability Limited Test Articles: Unit Cost, Production Time Requirements Validation: Do requirements satisfy mission requirements? Are they unambiguous, testable and technically realistic? Test Planning: Integrated test approach or serial? Active Participation in T&E WIPT? Early Identification of Operational Test Requirements? Test Execution: Operationally Representative Test Environments? Integrated Test Approach and Partnering Can be Improved

4. DOT&E Initiatives 4 Integrated Testing Developmental Operational Live Fire Improve Suitability Reliability Growth Testers Engage Early Requirements are Unambiguous Testable Relevant to mission accomplishment Technically Realistic Field New Capabilities Rapidly Accelerated Testing

5. Rapid Fielding Number One Priority of SECDEF Examples: –Extended Range Multi-Purpose UAS Early Fielding Report based on Customer Test –Project ODIN ATEC Observations Led to Improved Effectiveness –MRAP DoD 5000 Process Adapted for Rapid Development 55 ERMP UAS Counter IED OPS MRAP Live Fire Test Field New Capabilities Rapidly Accelerated Testing

6. Engage Early DOT&E Action Officer as a Team Member –Requirements Validation Do requirements satisfy mission requirements? Are they unambiguous, testable and technically realistic –Test Planning Active Participation in T&E WIPT Early Identification of Operational Test Requirements –Test Execution Operationally Representative Test Environments 6 Testers Engage Early Requirements are …

7. Integrated Testing Weapon System Acquisition Reform Act –Established Director of Developmental Test –Formally Implemented Integrated Testing Test Process as a Continuum –Developmental Testing –Live-fire Testing –Operational Testing Optimize Testing for Program Execution –Design of Experiments –Operational Test Evaluation Utilizes All Data 7 Integrated Testing Developmental Operational Live Fire

8. Improve Suitability After System Effectiveness... Suitability is the Key Concern 8 Improve Suitability Reliability Growth Systems frequently judged Un-Suitable Reliability and Maintainability account for 80% of Not Suitable ratings

9. Implementation (Continued) Integrated Testing –Test Process as a Continuum Developmental Testing Live-fire Testing Operational Testing –Use Statistical Analysis Data Collected over Long Period & Multiple Tests Design of Experiments Normalization of Data 9

10. 10 Common Activities Replicating the “real world” environment as closely as practical Need for a distributive live/virtual/constructive (LVC) representation of the Joint Operational Environment Development and use of validated Tactics, Techniques, and Procedures (TTPs) Development of COCOM/JS approved Joint Mission Threads Data collection, management, archiving, and retrieval processes The Test and Training Communities are Missing Opportunities for Cost-effective, Relevant Partnerships

11. 11 Common Activities Replicating the “real world” environment as closely as practical Need for a distributive live/virtual/constructive (LVC) representation of the Joint Operational Environment Development and use of validated Tactics, Techniques, and Procedures (TTPs) Development of COCOM/JS approved Joint Mission Threads Data collection, management, archiving, and retrieval processes The Test and Training Communities are Missing Opportunities for Cost-effective, Relevant Partnerships

12. 12 Bottom Line on What’s Needed Strong partnership between the COCOMS, Test, and Training communities Committed leadership for strategic institutional funding, co- investment, and cross-community management of joint test and training resources and infrastructure –Need to overcome preference for OPM (Other People’s Money) –Need to continue to work towards implementation of joint test and training roadmaps It Takes Time and Effort – but Potential for Increased Efficiencies and Effectiveness Justifies Both

13. Design of Experiments (DOE) Test planning is a science! DOE is a scientific tool for developing robust test plans. DOE equips us to determine: –Power –Confidence –Breadth of coverage The iterative nature of DOE facilitates Integrated Test design DOE Manage Plan Test Design Analyze

14. Design of Experiments Integrated Test – DOE Approach –Inform each stage of testing from previous tests. –Span the battlespace through a compilation of tests. –Use screening experiments in DT to ensure a rigorous OT. DOE Provides: –Most powerful allocation of test resources for a given number of tests. –A scientific, structured, objective way to plan tests. –An approach to integrated test. –A structured, mathematical analysis for summarizing test results. Developmental Test DT/OT Testing Operational Test 14

15. Design of Experiments:JASSM-ER Binomial Distribution was Primary Method to Establish JASSM-ER Integrated Test Design Resulted in 21-shot Design Assumes ≤ 2 failures, to Demonstrate 0.80 Reliability with 80% Confidence 24 Factors Considered; Refined Design by Reducing Factors Previously Evaluated in Baseline JASSM OT Primary Factors: Range, Release Mach, Release Altitude, & Target Type Ensures Key Performance Differences between Baseline JASSM & JASSM-ER are Evaluated 21-event Test Design Created Using ¾–fractional Design to Estimate Main Effects & Interactions with Range Uses 5 Combined DT/OT Shots 15

16. Conclusion Our Goals –Greater Rigor and Effectiveness in Testing –Introduce New Statistical Developments Results –Better Confidence, Power & Breadth in Tests Our Products –More Effective and Suitable Systems Rigor is Key to Constructing & Executing Efficient Tests Only Efficient Testing can be Defended in the Current and Future Budget Environment 16

17. Questions 17