NDIA SE Division Meeting February 12, Developmental Test and Evaluation Committee Status and Plans February 2014 Beth Wilson, Raytheon Steve Scukanec, Northrop Grumman Industry Co-Chairs

NDIA SE Division Meeting February 12, DT&E Committee 2014 Status and Plans ActivityPlans for 2014Status T&E Cyber TestingIndustry recommendations for cyber testing Draft complete Submit to DASD DT&E Feb Chief Developmental Tester Industry interaction with new roleWhite paper outline complete Soliciting ase study examples ReliabilityRecommendations for better reliability testing IEEE standards liaisons established T&E WorkshopSummer workshopTutorials and papers SE Modeling & SimulationT&E Perspective for Modeling/ Simulation products Joint meeting TBD ArchitectureT&E Perspective for Architecture views Joint meeting TBD Education & TrainingChief Developmental Tester role(after CDT project complete) Systems Security Engineering Cyber testing guidelines connections to PPP (after SSE PPP May workshop) SE ConferenceOctober conferenceDT&E track, joint tracks CompleteIn ProcessProposed

NDIA SE Division Meeting February 12, Cyber Testing Guidelines ActivityPlans for Status/Plans Cyber TestingReview what industry does for cyber testing, expand focus to include interoperability Recommendations for Cyber Testing guidelines Project Lead: Dave Desjardins Joint Project with ITEA

NDIA SE Division Meeting February 12, Chief Developmental Tester ActivityPlans for Status/Plans Chief Developmental Tester Review service policies for new role and industry implementations Propose model for industry interaction White paper T&E/SE conference presentations The Secretary of Defense shall require that each major defense acquisition program be supported by— ‘‘(A) a chief developmental tester; and ‘‘(B) a governmental test agency, serving as lead developmental test and evaluation organization for the program. Coordinating DT&E activities Insight into Contractor activities Oversee T&E activities Inform government PM about contractor DT&E results Project Lead: Joe Manas

NDIA SE Division Meeting February 12, Reliability Testing Reliability Testing Focus - Software reliability standard - Cyber testing initiatives - Data for reliability growth measures Input/Feedback Opportunities: - IEEE meetings January and August - Periodic revision periods ActivityPlans for 2013Status/Plans Reliability Testing (with IEEE) Develop a better test strategy for reliability (collaborate with IEEE Reliability Society) Solicit feedback on existing standards Identify recommendations to improve reliability testing Project Leads: Lou Gullo, Taz Daughtrey

NDIA SE Division Meeting February 12, DT&E Committee Task Plan Projects Working Group Proposed 2014 Tasks: Cyber Testing Project Chief Developmental Tester Project Collaboration with IEEE Reliability Joint meetings with SED Committees Modeling and Simulation Architecture Systems Security Engineering Education and Training Deliverables/Products Cyber Testing Recommendations white paper Chief Developmental Tester white paper Candidate KSABs for role (E&T) Industry interaction recommendation Tutorial on role for SE conference Inputs to reliability standards effort Joint meeting notes Schedule / Resources Cyber Testing Draft Recommendations Feb 2014 Weekly teleconferences started 12/3/2013 Full white paper later in 2014 Chief Developmental Tester Joint meeting with E&T with SED meeting Tutorial at SE conference Reliability standards input for Jan/Aug meetings Test as a Stakeholder joint meetings with SED Modeling and Simulation Architecture Systems Security Engineering Issues / Concerns: Industry engagement NDIA T&E conference may be cancelled (Seattle in March) ITEA conference (Denver in September) may be postponed and moved

NDIA SE Division Meeting February 12, Backup DT&E Committee Results

NDIA SE Division Meeting February 12, T&E Themes For Conferences SED Committee Collaborative Projects DT&E Committee Current Structure Since 2010 NDIA Systems Engineering Conference (Q4) Collaborative Projects NDIA Test and Evaluation Annual Conference (Q1) Periodic T&E Committee-Focused Conferences Cooperation with other NDIA Divisions NDIA Industrial Committee on Test and Evaluation (ICOTE) DT&E Committee Focus: T&E initiatives aligned with SE, DT&E T&E Themes & Issues Project Results DT&E Themes NDIA Test and Evaluation Division NDIA Systems Engineering Division NDIA DT&E Committee SE Themes & Issues

NDIA SE Division Meeting February 12, Summary of DT&E Committee Test and Evaluation Efforts TopicActivity DT&E C ommittee DoD T&E Policy Study2006 – 2008: Workshops and Study Report Improving T&E in the DoD Acquisition Process Integrated Testing2008 – 2010: Integrated Test Study NDIA presentations and tutorial ITEA journal article RFP Language for T&E2010 – 2011: Comments for Update to OSD Guide Incorporating T&E into DoD Acquisition Contracts Test Optimization2012: Statistical Optimization Conference Thread 2013: Scientific Test and Analysis Techniques report/articles Cyber TestingStarted 2013: Industry recommendations for cyber testing Chief Developmental TesterStarted 2013: Industry interaction with new role

NDIA SE Division Meeting February 12, Summary of DT&E Committee Systems Engineering Efforts SED CommitteeActivity SE Collaboration Software2009: SW T&E Summit Recommendations System of Systems2010 – 2011: T&E for SoS Workshop and Initiatives 2012: Final Report on Best Practices Model Performance Measurement 2012: Leading Indicators for T&E workshop 2013: Requirements Verification leading indicators presentation Modeling and Simulation 2011: Effective Use of Modeling and Simulation for T&E 2012: Distributed Model-Based Testing ReliabilityStarted 2013: Recommendations for better reliability testing

NDIA SE Division Meeting February 12, DoD T&E Policy Study August 2006: DT&E Committee Kickoff Policy Study: “Improving T&E in the DoD Acquisition Process” Industry T&E policy recommendations Workshops: August 2007 January 2008 Focus Areas: 1. Earlier contractor and tester involvement 2. Integrated DT/OT and DT operational relevance 3. Suitability April 2008: Report Summarized Results: 10 Findings 15 Recommendations

NDIA SE Division Meeting February 12, Integrated Testing (CT/DT/OT) Implementation Framework

NDIA SE Division Meeting February 12, Software Test and Evaluation Software Summit RFP Language How Much Testing is Enough? Lifecycle and End-to-End Software T&E Changing Paradigms Training & Competency Model Policy, Guidance, and Standards Tools, Automation, Methodologies, Process 2009

NDIA SE Division Meeting February 12, RFP Language Industry # Master # Reviewer Line # Comment and RationaleRecommended InputA/R/P Industry Comments for Update: Incorporating Test and Evaluation Into Department of Defense Acquisition Contracts Recommendations from SW Summit

NDIA SE Division Meeting February 12, Test and Evaluation for Systems of Systems 2010: Workshop 2009: “Sleepless Nights” List of Issues 2010: “Sominex” Resulting Initiatives 2011: Best Practices Wave Model : Final Report

NDIA SE Division Meeting February 12, Effective Use of Modeling and Simulation for Test and Evaluation Joint Meeting in August 2011 Distributed Testing, the Joint Mission Environment Test Capability (JMETC) and the Test and Training Enabling Architecture (TENA) DoD M&S Community of Interest Data Management Working Group LVC Architecture Roadmap Implementation (LVCAR ‐ I) Gateways Effort Applicability to T&E OSD T&E Working Group Raytheon Presentation on M&S for T&E Potential Topics for November AMSWG Meeting 2011

NDIA SE Division Meeting February 12, Modeling & Simulation for Distributed Testing Benefits: Find integration issues earlier Test to learn in ‘safe’ environment Protect proprietary information Facilitate DT to OT transition Increase performance testing range in operating environments Support end to end studies throughout the program Barriers: Security Lack of persistent network Early consideration of technical issues Perceived value Disconnect between the communities (M&S and T&E) Recommendations: Harmonize the standards for M&S and Test for the life cycle perspective (HLA, TENA, Metadata) Create a framework for reusing and repurposing M&S through the product model Establish M&S as part of statistical test design Determine what tests are conducted to acquire data for model validation. Fewer test events with better models. Recommend the use of M&S to do I&T Recommend establishment of JMETC as a persistent node for industry to engage in MBDI&T Joint Track SE Conference October 2012 Joint Meeting August

NDIA SE Division Meeting February 12, Statistical Test Optimization 2012 SE Conference summit/workshop thread -Tutorials on Monday 10/22 - Presentations on Wednesday 10/24 - Synthesis Panel on Wednesday 10/24 Follow-on Results: - DT&E Committee white paper - ITEA Journal article - SE Conference track Oct MORS Conference Nov

NDIA SE Division Meeting February 12, Requirements Verification Leading Indicator Metrics 2012 SE Conference workshop - Follow-on workshop with System Performance Measurement WG - Focused on Requirements Verification not addressed in first report Potential Measures Identified: - System Maturity Level - Verification Requirement Maturity - Technical Measures and Stakeholder Need