1. DT&E Strategy and the Developmental Evaluation Framework (DEF) Concept & Program Implementation
5 June 2014

2. Discussion Topics
Concept – DT&E Strategy focused on evaluation and informing decisions
How to Reflect in TEMP – Framework format
How to Build – Core Team implementation

3. DT&E Strategy Overview
Articulate a logical evaluation strategy that informs decisions
How acquisition, programmatic, technical and operational decisions will be informed by evaluation
How system will be evaluated
How test and M&S events will provide data for evaluation
What resources are required to execute test, conduct evaluation, and inform decisions
Decisions
Evaluation
Test / M&S
Resources
Schedule
Define
Inform
Data
Execute
DT&E story thread: decision – evaluation– test & resources

4. Developmental Evaluation Framework (Defense Acquisition Guidebook)
Developmental Evaluation Framework (Enclosure 4, DoD Interim Instruction )
Test and Evaluation Master Plan (TEMP) includes a Developmental Evaluation Framework (“T&E Roadmap”)
Knowledge gained from testing provides information for technical, programmatic, and acquisition decisions.
DoDI (Interim)
Developmental Evaluation Framework:
Identifies key data that contributes to assessing progress on:
Key Performance Parameters
Critical Technical Parameters
Key System Attributes
Interoperability requirements
Cybersecurity requirements
Reliability growth
Maintainability attributes
Developmental test objectives
Others as needed
Show the correlation/mapping between:
Test events
Key resources
Decision supported
Developmental Evaluation Framework
(Defense Acquisition Guidebook)

5. Developmental Evaluation Framework (DEF)
Capability questions
Decision Support Questions (DSQ)
DEO 1
DEO 2
DEO 3
TM 1
TM 2
*TM 3
System capabilities
KPP/KSA/CTP -related
Technical measures
System Engineering decomposition:
Evaluate system capability - Inform decisions

6. The TEMP’s DT&E Strategy Story
TEMP tells the decision – evaluation – test/M&S story
Section 3.1 – T&E Strategy. Describe how T&E informs Acquisition Strategy decisions
Figure to accompany verbiage: Decision Support Key (DSK)
Describes decisions and T&E information needed
Section 3.3 – Developmental Evaluation Approach. Describe how system will be evaluated to inform decisions
Figure to accompany verbiage: Developmental Evaluation Framework (DEF)
Links decisions – evaluation – test/M&S events
Section 3.6 – Operational Evaluation Approach.
Decision Support Key (DSK) & Developmental Evaluation Framework (DEF) built by Chief Dev Tester
DEF Core Team is subgroup of T&E WIPT including Chief Developmental Tester and select SME’s

7. DT&E Informed Decisions
Acquisition
Technical
Programmatic
Operational
Acquisition decisions are those generally associated with execution of the DoD acquisition system. These are knowledge points called out in the 5000 series regulations and instructions. Example decision points where DT&E information would be relevant, are listed below:
Acquisition:
Readiness for EMD
Readiness for production
Readiness for full-rate production
Contract award(s), including long lead/advance procurement and LRIP buys
Programmatic decisions are those associated with execution of a specific program. These are the knowledge points that would have limited relevance to other systems or programs. Programmatic knowledge points may be used to support acquisition decisions. Example decision points where DT&E information would be relevant, are listed below:
Programmatic:
RFP content definition
Source selection criteria definition and evaluation(s)
Award fee criteria and determination(s)
PPBE
Whereas the acquisition and programmatic decisions are more focused on the business aspects of acquiring new capabilities, technical decisions are those associated with development of a specific system. These are design-related knowledge points that would have limited relevance to other systems or programs. Technical knowledge points may be used to support acquisition and programmatic decisions. Example decision points where DT&E information would be relevant, are listed below:
Technical:
Technology maturity
Technical (system) maturity (e.g. reliability growth, cybersecurity)
Prototype
Technical off-ramps
Correction of deficiencies
First flight
Ferry flight
Deployment
Certifications – Airworthiness, ATO, ATC, etc.
Informing decisions throughout lifecycle:
Same DEF concept/form; Different decisions and system info

8. Decision Support Key
Show how decisions will be informed by answering T&E focus questions: DT&E Decision Support Questions (DSQs) and OT&E Critical Operational Issues (COIs)
TEMP Section 3.1 – T&E Strategy – Describes how program’s Acquisition Strategy is informed by T&E Strategy

9. Developmental Evaluation Framework
Decisions
Evaluation
Test / M&S
Resources
Schedule
Define
Inform
Data
Execute

10. Link Resources & Schedule
Link key resources and schedule to DEF
Describe logical linkage of test/M&S events to necessary resources in Section IV
Describe linkage of decisions, evaluation, test, and M&S events to schedule in programmatic schedule in Section II

11. Core Teams: Assist Chief Developmental Tester in Applying DEF to Programs
More coherent and deliberate DT&E - needs a consistent DT&E strategy development process
DASD(DT&E)-led DEF Core Team
Small, focused group of SE, T&E and program acquisition SMEs
Chief Developmental Tester, Chief Engineer, Program Manager, Lead Developmental Test & Evaluation Organization lead, acquisition strategy SME, requirements SME
Develop DT&E strategy by facilitated discussion
Decisions Supported – Key program decisions needing T&E generated knowledge to inform them
Decision Support Questions (DSQ) - Questions related to performance, reliability, cybersecurity or interoperability that when answered provide T&E information to decision-makers
Developmental Evaluation Objectives (DEO) – system capability categories
Technical Measures (TM) – “inch deep-mile wide” quantification of capabilities
T&E and M&S events – data generators for system evaluation

12. Before the Core Team - Gather the People and Documents
Identify small, focused group members, including:
DASD(DT&E) Staff Specialist
Educate on concept, lead & facilitate strategy development
Chief Developmental Tester
Take ownership of Developmental Evaluation Strategy
Program Manager (or representative)
Acquisition Strategy SME – to define decisions supported
Program Chief Engineer, Requirements SME (DOORS™ expert)
Capability / requirements SME – to define capabilities and measures
Lead Developmental T&E Organization representative
Test, M&S SME – to define test and M&S events
Gather needed reference material, schedule ~2 days
DASD(DT&E): DT&E Strategy concept & example presentation
Program office:
Acquisition strategy & schedule
Technical & ops requirements: JCIDS (ICD, CDD, CPD), TRD, SRD, A-spec, SEP, PPP, and/or DOORS requirements decomposition
OTA’ s OT&E evaluation framework (if available)

13. The Core Team Process 1st - Defining the Decisions
Identify key decisions & information needs
Identify programmatic, acquisition or technical decisions (PM, PEO, AT&L) throughout acquisition strategy
Articulate essence of decision being made
Determine DT&E information needed to support decision-making in form of Decision Support Questions (DSQ)
Document in Decision Support Key

14. The Core Team Process 2nd - Defining the System Capabilities
Develop Developmental Evaluation Objectives (DEO)
System technical capabilities
Binned into Functional Evaluation Areas (performance, interoperability, cybersecurity, reliability)
Suggested starting point: major TRD paragraph headings
Expand or contract to generate top-level listing of technical capabilities

15. The Core Team Process 3rd - Defining the Measures
Determine appropriate level or detail for measures
Quantify “inch-deep & mile-wide” measures
Not just KPP/KSA/CTPs
Depending upon number of tech requirements:
Technical measures
TRD sub-paragraph (binning of several related measures)
DOORS hierarchy-cut
Binning of related measures

16. The Core Team Process 4th - Defining the “Important” Measures
Highlight important technical measures for additional evaluation emphasis
Designate (bold & asterisk) KPP/KSA/CTP – related measures

17. The Core Team Process 5th - Defining Test & M&S data sources
Define T&E and M&S events
Identify T&E and/or M&S events that will generate data for evaluation
Identify system security engineering activities (in PPP)
Place event nomenclature at cell at intersection of decision and capability/measure

18. Informed decisions: right information – right time – right person
Summary
DT&E Strategy and Developmental Evaluation Framework (DEF) tells logical story of how DT&E informs decisions throughout acquisition
DASD(DT&E) expects Developmental Evaluation Framework in TEMPs
Not going to leave you hanging!
Assistance available to implement in your program
Standardized concept intro brief & DAG guidance
Training / assistance with Core Team leadership
Informed decisions: right information – right time – right person

19. Questions?
Dr. Suzanne Beers