DT&E Strategy and the Developmental Evaluation Framework (DEF) Concept & Program Implementation 5 June 2014

Discussion Topics Concept – DT&E Strategy focused on evaluation and informing decisions How to Reflect in TEMP – Framework format How to Build – Core Team implementation

DT&E Strategy Overview Articulate a logical evaluation strategy that informs decisions How acquisition, programmatic, technical and operational decisions will be informed by evaluation How system will be evaluated How test and M&S events will provide data for evaluation What resources are required to execute test, conduct evaluation, and inform decisions Decisions Evaluation Test / M&S Resources Schedule Define Inform Data Execute DT&E story thread: decision – evaluation– test & resources

Developmental Evaluation Framework (Defense Acquisition Guidebook) Developmental Evaluation Framework (Enclosure 4, DoD Interim Instruction 5000.02) Test and Evaluation Master Plan (TEMP) includes a Developmental Evaluation Framework (“T&E Roadmap”) Knowledge gained from testing provides information for technical, programmatic, and acquisition decisions. DoDI 5000.02 (Interim) Developmental Evaluation Framework: Identifies key data that contributes to assessing progress on: Key Performance Parameters Critical Technical Parameters Key System Attributes Interoperability requirements Cybersecurity requirements Reliability growth Maintainability attributes Developmental test objectives Others as needed Show the correlation/mapping between: Test events Key resources Decision supported Developmental Evaluation Framework (Defense Acquisition Guidebook)

Developmental Evaluation Framework (DEF) Capability questions Decision Support Questions (DSQ) DEO 1 DEO 2 DEO 3 TM 1 TM 2 *TM 3 System capabilities KPP/KSA/CTP -related Technical measures System Engineering decomposition: Evaluate system capability - Inform decisions

The TEMP’s DT&E Strategy Story TEMP tells the decision – evaluation – test/M&S story Section 3.1 – T&E Strategy. Describe how T&E informs Acquisition Strategy decisions Figure to accompany verbiage: Decision Support Key (DSK) Describes decisions and T&E information needed Section 3.3 – Developmental Evaluation Approach. Describe how system will be evaluated to inform decisions Figure to accompany verbiage: Developmental Evaluation Framework (DEF) Links decisions – evaluation – test/M&S events Section 3.6 – Operational Evaluation Approach. Decision Support Key (DSK) & Developmental Evaluation Framework (DEF) built by Chief Dev Tester DEF Core Team is subgroup of T&E WIPT including Chief Developmental Tester and select SME’s

DT&E Informed Decisions Acquisition Technical Programmatic Operational Acquisition decisions are those generally associated with execution of the DoD acquisition system. These are knowledge points called out in the 5000 series regulations and instructions. Example decision points where DT&E information would be relevant, are listed below: Acquisition: Readiness for EMD Readiness for production Readiness for full-rate production Contract award(s), including long lead/advance procurement and LRIP buys   Programmatic decisions are those associated with execution of a specific program. These are the knowledge points that would have limited relevance to other systems or programs. Programmatic knowledge points may be used to support acquisition decisions. Example decision points where DT&E information would be relevant, are listed below: Programmatic: RFP content definition Source selection criteria definition and evaluation(s) Award fee criteria and determination(s) PPBE Whereas the acquisition and programmatic decisions are more focused on the business aspects of acquiring new capabilities, technical decisions are those associated with development of a specific system. These are design-related knowledge points that would have limited relevance to other systems or programs. Technical knowledge points may be used to support acquisition and programmatic decisions. Example decision points where DT&E information would be relevant, are listed below: Technical: Technology maturity Technical (system) maturity (e.g. reliability growth, cybersecurity) Prototype Technical off-ramps Correction of deficiencies First flight Ferry flight Deployment Certifications – Airworthiness, ATO, ATC, etc. Informing decisions throughout lifecycle: Same DEF concept/form; Different decisions and system info

Decision Support Key Show how decisions will be informed by answering T&E focus questions: DT&E Decision Support Questions (DSQs) and OT&E Critical Operational Issues (COIs) TEMP Section 3.1 – T&E Strategy – Describes how program’s Acquisition Strategy is informed by T&E Strategy

Developmental Evaluation Framework Decisions Evaluation Test / M&S Resources Schedule Define Inform Data Execute

Link Resources & Schedule Link key resources and schedule to DEF Describe logical linkage of test/M&S events to necessary resources in Section IV Describe linkage of decisions, evaluation, test, and M&S events to schedule in programmatic schedule in Section II

Core Teams: Assist Chief Developmental Tester in Applying DEF to Programs More coherent and deliberate DT&E - needs a consistent DT&E strategy development process DASD(DT&E)-led DEF Core Team Small, focused group of SE, T&E and program acquisition SMEs Chief Developmental Tester, Chief Engineer, Program Manager, Lead Developmental Test & Evaluation Organization lead, acquisition strategy SME, requirements SME Develop DT&E strategy by facilitated discussion Decisions Supported – Key program decisions needing T&E generated knowledge to inform them Decision Support Questions (DSQ) - Questions related to performance, reliability, cybersecurity or interoperability that when answered provide T&E information to decision-makers Developmental Evaluation Objectives (DEO) – system capability categories Technical Measures (TM) – “inch deep-mile wide” quantification of capabilities T&E and M&S events – data generators for system evaluation

Before the Core Team - Gather the People and Documents Identify small, focused group members, including: DASD(DT&E) Staff Specialist Educate on concept, lead & facilitate strategy development Chief Developmental Tester Take ownership of Developmental Evaluation Strategy Program Manager (or representative) Acquisition Strategy SME – to define decisions supported Program Chief Engineer, Requirements SME (DOORS™ expert) Capability / requirements SME – to define capabilities and measures Lead Developmental T&E Organization representative Test, M&S SME – to define test and M&S events Gather needed reference material, schedule ~2 days DASD(DT&E): DT&E Strategy concept & example presentation Program office: Acquisition strategy & schedule Technical & ops requirements: JCIDS (ICD, CDD, CPD), TRD, SRD, A-spec, SEP, PPP, and/or DOORS requirements decomposition OTA’ s OT&E evaluation framework (if available)

The Core Team Process 1st - Defining the Decisions Identify key decisions & information needs Identify programmatic, acquisition or technical decisions (PM, PEO, AT&L) throughout acquisition strategy Articulate essence of decision being made Determine DT&E information needed to support decision-making in form of Decision Support Questions (DSQ) Document in Decision Support Key

The Core Team Process 2nd - Defining the System Capabilities Develop Developmental Evaluation Objectives (DEO) System technical capabilities Binned into Functional Evaluation Areas (performance, interoperability, cybersecurity, reliability) Suggested starting point: major TRD paragraph headings Expand or contract to generate top-level listing of technical capabilities

The Core Team Process 3rd - Defining the Measures Determine appropriate level or detail for measures Quantify “inch-deep & mile-wide” measures Not just KPP/KSA/CTPs Depending upon number of tech requirements: Technical measures TRD sub-paragraph (binning of several related measures) DOORS hierarchy-cut Binning of related measures

The Core Team Process 4th - Defining the “Important” Measures Highlight important technical measures for additional evaluation emphasis Designate (bold & asterisk) KPP/KSA/CTP – related measures

The Core Team Process 5th - Defining Test & M&S data sources Define T&E and M&S events Identify T&E and/or M&S events that will generate data for evaluation Identify system security engineering activities (in PPP) Place event nomenclature at cell at intersection of decision and capability/measure

Informed decisions: right information – right time – right person Summary DT&E Strategy and Developmental Evaluation Framework (DEF) tells logical story of how DT&E informs decisions throughout acquisition DASD(DT&E) expects Developmental Evaluation Framework in TEMPs Not going to leave you hanging! Assistance available to implement in your program Standardized concept intro brief & DAG guidance Training / assistance with Core Team leadership Informed decisions: right information – right time – right person

Questions? Dr. Suzanne Beers sbeers@mitre.org 719-572-8257