Guidance for Writing: Test & Evaluation Master Plans TEMP Preparation 101 Brief September 2009

What is a TEMP ? ‘Contract’ Between All Stakeholders! Executive level strategy and basic planning document for all system life cycle T&E Overall T&E structure, major elements, and objectives Consistent with: Acquisition Strategy Technology Development Strategy (TDS) Capability Development / Production Document (CDD / CPD) System Threat Assessment Information Support Plan ( ISP) Consistent and complementary to the Systems Engineering Master Plan Sufficient detail to permit planning for timely availability of test resources required to support the T&E program Road Map for: Evaluation framework Resource requirements Schedules and decisions supported ‘Contract’ Between All Stakeholders!

Policy / Guidance OSD: Army: DOD Directive 5000.01, The Defense Acquisition System, 12 May 2003 DOD Instruction 5000.02: Operation of the Defense Acquisition System, 8 Dec 2008 Interim Defense Acquisition Guidebook, 7 July 2009 Army: AR 73-1, Test and Evaluation Policy, 1 August 2006 Chapter 8: T&E Working-level Integrated Product Team (T&E WIPT) Chapter 10: T&E Review and Reporting Requirements - Paragraph 10-2. T&E Master Plan (TEMP) DA PAM 73-1, T&E in Support of Systems Acquisition, 30 May 2003 Table of Contents OSD documents are accessible at DAU web site http://akss.dau.mil/ and then using the Search engine Army documents are accessible via Army T&E page on AKO website: https://www.hqda.army.mil/suite/page/25

Materiel and Information Technology Applicable Programs Every Army acquisition program will have a TEMP Only exceptions are: Investigational drugs Biologicals Medical devices Includes Materiel and Information Technology Programs

Preparation Responsibility: Program / Project Manager (PM) Execution: T&E WIPT (DA Pamphlet 73-1, Chapter 2)

TEMP Content OSD TEMP Approval Memo Approval Page Cover Sheet T&E WIPT Coordination Sheet Table of Contents Part I - Introduction Part II - Test Program Management and Schedule Part III - Test and Evaluation Strategy Part IV - Resource Summary Annex A - Bibliography Annex B - Acronyms Annex C - Points of Contact Attachment 1 – Requirements / Test Crosswalk Matrix Additional appendices / annexes / attachments as needed

TEMP Part I (Introduction) Preparation Responsibility Matrix 1 of 5 TEMP PART & SECTION PM SE CD / FP TI LOG T&E ACTIVITY 1.1 Purpose P S 1.2 Mission Description 1.3 System Description 1.3.1 System Threat Assessment 1.3.2 Program Background 1.3.3 Key Capabilities Legend P: Primary Responsibility PM: Program Manager CD/FP: Combat Developer/Functional Proponent S: Support Responsibility SE: Systems Engineer TI: Threat Integrator LOG: Logistician 7

TEMP Part II (Test Program Management & Schedule) Preparation Responsibility Matrix 2 of 5 TEMP PART & SECTION PM SE CD / FP TI LOG T&E ACTIVITY 2.1 T&E Management P S 2.2 Common T&E Data Base Requirements 2.3 Deficiency Reporting 2.4 TEMP Updates 2.5 Integrated Test Program Summary Legend P: Primary Responsibility PM: Program Manager CD/FP: Combat Developer/Functional Proponent S: Support Responsibility SE: Systems Engineer TI: Threat Integrator LOG: Logistician 8

TEMP Part III (T&E Strategy) Preparation Responsibility Matrix 3 of 5 TEMP PART & SECTION PM SE CD / FP TI LOG T&E ACTIVITY 3.1 T&E Strategy P 3.2 Evaluation Framework S 3.3 Developmental Evaluation Approach 3.4 Live Fire Evaluation Approach 3.5 Certification for Initial Operational Test 3.6 Operational Evaluation Approach 3.7 Other Certifications 3.8 Reliability Growth 3.9 Future T&E Legend P: Primary Responsibility PM: Program Manager CD/FP: Combat Developer/Functional Proponent S: Support Responsibility SE: Systems Engineer TI: Threat Integrator LOG: Logistician 9

TEMP Part IV (Resource Summary) Preparation Responsibility Matrix 4 of 5 TEMP PART & SECTION PM SE CD / FP TI LOG T&E ACTIVITY 4.1 Introduction 4.1.1 Test Articles P S 4.1.2 Test Sites & Instrumentation 4.1.3 Test Support Equipment 4.1.4 Threat Representation 4.1.5 Test Targets and Expendables 4.1.6 Operational Force Test Support 4.1.7 Models, Simulations, & Testbeds 4.1.8 Joint Operational Test Environment 4.1.9 Special Requirements 4.2 Federal, State, & Local Requirements 4.3 Manpower / Personnel Training 4.4 Test Funding Summary Table Resource Summary Table Legend P: Primary Responsibility PM: Program Manager CD/FP: Combat Developer/Functional Proponent S: Support Responsibility SE: Systems Engineer TI: Threat Integrator LOG: Logistician 10

TEMP Appendices / Attachments Preparation Responsibility Matrix 5 of 5 TEMP PART & SECTION PM SE CD / FP TI LOG T&E ACTIVITY Appendix A. Bibliography P S Appendix B. Acronyms Appendix C. Points of Contact Attachment 1. Requirements / Test Crosswalk Matrix Other Annexes / Attachments Legend P: Primary Responsibility PM: Program Manager CD/FP: Combat Developer/Functional Proponent S: Support Responsibility SE: Systems Engineer TI: Threat Integrator LOG: Logistician 11

Content Detail NOTE Following TEMP Sections will be populated with actual excerpts from 4-Part TEMPs approved by the Office of Secretary of Defense.

TEMP Content Detail, cont. Part I. Introduction 1.1 Purpose 1.2 Mission Description 1.3 System Description 1.3.1 System Threat Assessment 1.3.2 Program Background 1.3.3 Key Capabilities

TEMP Content Detail, cont. Part II. Test Program Management & Schedule 2.1 T&E Management 2.2 Common T&E Data Base Requirements 2.3 Deficiency Reporting 2.4 TEMP Updates 2.5 Integrated Test Program Summary

TEMP Content Detail, cont. Part III. Test & Evaluation Strategy 3.1 T&E Strategy 3.2 Evaluation Framework 3.3 Developmental Evaluation Approach 3.4 Live Fire Evaluation Approach 3.5 Certification for Initial Operational Test 3.6 Operational Evaluation Approach 3.7 Other Certifications 3.8 Reliability Growth 3.9 Future T&E

TEMP Content Detail, cont. Part IV. Resource Summary 4.1 Introduction 4.1.1 Test Articles 4.1.2 Test Sites & Instrumentation 4.1.3 Test Support Equipment 4.1.4 Threat Representation 4.1.5 Test Targets and Expendables 4.1.6 Operational Force Test Support 4.1.7 Models, Simulation, & Testbeds 4.1.8 Joint Operational Test Environment 4.1.9 Special Requirements 4.2 Federal, State, and Local Requirements 4.3 Manpower / Personnel Training 4.4 Test Funding Summary Table 4.1 Resource Summary Matrix

TEMP Content Detail, cont. Appendix A. Bibliography Appendix B. Acronyms Appendix C. Points of Contact Attachment 1. Requirements / Test Crosswalk Matrix Other Annexes / Attachments

TEMP - SEP CONTRAST TEMP SEP SEP TEMP T&E Strategy Integrated T&E WIPT produces MDA (or designee) approves Serves as a T&E ‘Contract’ Issue level of detail Can include all events ATEC System Team (AST) produces ATEC Commander (or designee) approves Outlines HOW evaluation will be done Measure level of detail Only events necessary for evaluation Addresses Test Plan(s) SEP TEMP T&E Strategy Integrated While ATEC’s SEP and the T&E WIPT’s TEMP are DEVELOPED CONCURRENTLY, it is IMPORTANT to know the SALIENT DIFFERENCES between the two documents. GO OVER EACH ITEM. Developed concurrently, BOTH documents are CRUCIAL to a program’s successful Integrated T&E process. Some day you may deal with a PM (or T&E WIPT Chair) named ‘Bucky.’ AR 73-1 (para 1-4h) – ‘The detailed information contained in the SEP supports concurrent development of the TEMP.’ 18

T&E WIPT Coordination Process * Army TEMP T&E WIPT Coordination Process * INPUT Acquisition Strategy Requirements Documents: CDD / CPD System Threat Assessment Critical Operational Issues and Criteria Technology Development Strategy (TDS) Information Support Plan (ISP) Combat Developer PM T&E Manager Training Developer T&E WIPT Meetings ** T&E WIPT Draft TEMP Sent to T&E WIPT Other Agencies HQDA ASA(ALT) ASA(ALT) ILS DUSA-TE DCS, G-1 DCS, G-2 DCS, G 3 / 5 / 7 CIO / G-6 DCS, G-8 OSD Action Officers SMDC ATEC T&E WIPT TEMP Concerns/Feedback T&E Working-level IPT needs to resolve all T&E issues BEFORE signing the TEMP’s ‘T&E WIPT Coordination Sheet!’ Final T&E WIPT ‘Signing Party’ DTC OTC AEC * T&E Working-level Integrated Product Team ** Meetings repeated until TEMP coordination complete T&E WIPT Coordinated TEMP Source: Adapted from DA Pamphlet 73 - 1, Figure 3 - 1 19

TEMP Staffing and Approval Process Army Programs T&E WIPT & HQDA Staff Coordinated TEMP ACAT I and II • PEO or Developing Yes DUSA-TE Staffs Program Manager OSD T&E Oversight Agency Signs (if necessary) Signs & Submits or • TRADOC Signs Army Special • ATEC Signs Interest GOAL: Simultaneous Concurrent Staffing Army T&E Executive Approves No DUSA-TE Forwards EMPHASIZE: Process – Army Only T&E Executive’s Role Army T&E Executive sole signature Process slightly different for Multi-Service TEMPs: Army Lead & Army Participating Program Manager and T&E WIPT MDA OSD T&E Oversight or designated TEMP DUSA-TE Distributes approval authority for ACAT III program TIMELINES OSD Approves - D, DT&E * PM Signature to Army T&E Executive Signature - 30 Calendar Days or less OSD TEMP Approval Objective - 45 Calendar Days or less - D, OT&E Source: Adapted from DA Pamphlet 73 – 1, Figure 3 - 2 * ASD(NII) if space, IT, or C3 systems 20

TEMP Staffing and Approval Process Multi-Service OSD T&E Oversight Programs – Army Lead ACAT I or OSD T&E Oversight Signatures PEO or Developing Agency TRADOC ATEC Participating Service: OTA PEO or Dev Agency User Rep T&E WIPT & HQDA Staff Coordinated TEMP PM Submits Yes DUSA-TE Forwards Other Service Approvals No Army T&E Executive Approval MDA * Approval TRADOC ATEC T&E WIPT Other Services DUSA-TE Forwards OSD Approval D, DT&E - D, OT&E Program Manager DUSA-TE Distributes TIMELINES PM Signature to Army T&E Executive Approval Signature – 30 Calendar Days or less OSD Signature Objective – 45 Calendar Days or less * ACAT II programs having MDA delegation also have TEMP delegated approval authority Source: DA Pam 73-1, Figure 3-4, and Interim Defense Acquisition Guidebook, 7 July 2009

TEMP Staffing and Approval Process Multi-Service OSD T&E Oversight Programs – Army Participant ACAT I or OSD T&E Oversight T&E WIPT Equivalent Coordinated TEMP Army PM, PEO or Developing Agency Signs TRADOC Signs ATEC Signs Yes Lead Service Acquisition Executive DUSA-TE Staffs Army T&E Executive Approves No HQDA Staff Coordination Lead Service Acquisition Executive TRADOC ATEC T&E WIPT Other Services OSD Approval D, DT&E - D, OT&E Lead Service Acquisition Executive Army PM, PEO or MDA DUSA-TE Distributes TIMELINES TEO Receipt to Army T&E Executive Approval Signature – 20 Calendar Days or less OSD Signature Objective – 45 Calendar Days or less Source: DA Pam 73-1, Figure 3-5, and Interim Defense Acquisition Guidebook, 7 July 2009

TEMP Staffing and Approval Process Non-OSD T&E Oversight – ACAT II, III, & Army Special Interest Programs Program Manager and T&E WIPT PEO (if not MDA) TRADOC ATEC & HQDA Staff Coordinated TEMP Program Manager Signs ACAT II PM Forwards MDA or Designated TEMP Approval Authority ACAT III and Non-Tactical C4/IT Programs

Narrative TEMP Approval Process (TEMPs requiring HQDA and/or OSD approval) T&E WIPT develops TEMP (all members empowered; includes HQDA Staff) Final T&E WIPT meeting (all members sign TEMP coordination sheet) PM signs TEMP Approval Page; obtains PEO, TRADOC & ATEC simultaneous concurrence (Goal: Complete at T&E WIPT via empowered members) PM delivers TEMP with all concurrence signatures to DUSA-TE DUSA-TE staffs TEMP, as req’d; delivers to ADUSA(T&E) for Army approval (DUSA-TE staffing includes HQDA staff when not done during T&E WIPT staffing) ADUSA(T&E) provides approval signature within 10 days DUSA-TE sends TEMP to OSD (DDR&E, D,DT&E (Army goal is within 30 calendar days from PM submission signature) OSD staffing and approval objective (DDR&E, D,DT&E (or DASD C3ISR & IT Acq) sends to D,OT&E. (45 calendar days from receipt to return to TEO) NOTE: TEMPs are first required at milestone / key decision point entry into the acquisition life cycle. TEMPs are required to be updated at Milestone C and the Full Rate Production Decision Review, but should also be updated when the program baseline has been breached, when the associated Joint Capabilities Integration and Development System document or Information Support Plan has been significantly modified, or on other occasions when the program is significantly changed or restructured.

Ten TEMP “Do’s” and “Don’ts” (SOURCE: TEMAC T&E WIPT ‘Smart Book’) 1. Do ensure Core T&E WIPT members agree with any change(s) required to obtain TEMP approval after providing their coordination signature. 2. Do enforce configuration control over the TEMP. 3. Do have Part IV (Resources) based upon planned testing. 4. Do have the Requirements / Test Crosswalk Matrix as Attachment 1. 5. Do know the TEMP Approval Authority before developing the TEMP. 6. Do be very aware of COIC & CDD / CPD staffing approval timelines. 7. Do treat the TEMP as a CONTRACT among all Stakeholders!!! 8. Don’t attempt to gain TEMP approval before obtaining COIC approval. 9. Don’t “Concur With Comment” on TEMP’s T&E WIPT Coordination Sheet. 10. Don’t have TEMP simply refer to the System Evaluation Plan (SEP). Leave you with these points. Keep each in mind as you execute your ATEC System Team (AST) member RESPONSIBILITIES. Discuss each [if time permits.] Eight HQDA Offices: TEMA for the Army T&E Executive ASA(ALT) has 2 (ILS Division & Sys Mgmnt Integration) CIO/G-6; G-1, G-2, G-3/5/7, G-8

“Deadly Sins” Affecting TEMP Approval (SOURCE: TEMAC T&E WIPT ‘Smart Book’) Premature Submission: Submitting a TEMP for approval when an issue(s) precluding TEMP approval exists Failure to Pre-Coordinate: HQDA: Deputy Under Secretary of the Army for T&E (DUSA-TE) and HQDA Staff OSD: Director, DT&E and Director, OT&E Incomplete: Requirements / Test Crosswalk Matrix missing Classified data not integrated - submitted separately Too many TBDs in Resources Part COIC: Not HQDA approved and incorporated Size: Body of plan, excluding annexes, not kept to a manageable length Annexes: Not kept to a minimum Reference Documents: ICD / CDD / CPD & STAR not approved / unavailable

T&E WIPT “Do’s” and “Don’ts” (SOURCE: TEMAC T&E WIPT ‘Smart Book’) Do invite HQDA offices to participate in the T&E WIPT Do obtain appropriate OSD office(s) involvement early Do know T&E WIPT issues & be empowered Do hold program to the rigorous T&E process Do base positions on facts, vice emotions Do elevate an issue if unable to resolve within the T&E WIPT Don’t dismiss T&E WIPT meetings as unimportant Don’t dismiss T&E WIPT members’ comments without reasoned debate Don't allow unresolved issues to fester within the T&E WIPT (that is, take extra effort to resolve one-on-one & face-to-face) Don’t disband the T&E WIPT after TEMP approval Can apply to ASTs, too!

OSD TEMP Process

OSD T&E Oversight Programs TEMP Signature Process * SECDEF AT&L NII D, OT&E (OT TEMP CO- APPROVAL W/ D, DT&E) “ADVISE” “ADVISE” USD(AT&L) DDR&E Director, DT&E (DT TEMP Approval) C3ISR & IT Acquisition (DT TEMP Approval) DOT&E AO TEMP@OSD.mil or TEMP@OSD.smil.mil * Non-CBDP TEMPs DUSA-TE Forwards HQDA Approved TEMP

OSD TEMP Process, cont. Develop the TEMP through the T&E WIPT process Add the DDT&E / DOT&E action officers to your T&E WIPT Maintain informal dialog with the DDT&E and DOT&E action officers, to include contractor support Use Defense Acquisition Guidebook for TEMP development Provide sufficient details about future T&E activities (Part III, Sec 3.9) Be specific while eliminating the “fluff” TEMP is a master strategy document, not a detailed test plan. TEMP answers the “how” not the “why”

OSD TEMP Process, cont. Work with the Defense Intelligence Agency action officers Plan to test the system against the threats in the System Threat Assessment To reduce TEMP size, use pointers to other documents where more details are present, i.e.: Software development plans, Modeling and simulation plans Logistic support plans Include Modeling & Simulation in the T&E planning (Part IV, Sec 4.1.7) Reduce cycle time & costs

OSD TEMP Process, cont. Work to integrate / combine DT and OT to achieve mutual objectives at reduced costs Successfully completing independent OT&E (Title 10 statute requirement) is the “final exam” in T&E planning Be careful to not give up control of the DT&E effort to the OT&E There will still be a requirement for a dedicated OT&E event that the OTA controls DT&E is part of the risk reduction measures that are needed to ensure the system is ready for operational testing If the size and scope of the dedicated OT&E can be reduced without jeopardizing DT&E objectives, then integrating / combining DT and OT makes sense

OSD TEMP Process, cont. Submit electronic version of draft TEMP to OSD for staff review Identify “showstoppers” before TEMP enters the Service formal staffing and approval process Let DDT&E and DOT&E know early if an extension to the TEMP Due Date is needed Final staffing at OSD is a 45-day window for DDT&E and DOT&E signature

Request Your Feedback Please contact - Deputy Under Secretary of the Army for T&E 102 Army Pentagon Washington, D.C. 20310-0102 (703) 602-3508 / 2134 DSN: 332