Developmental Test and Requirements: Best Practices of Successful Information Systems using Agile Methods LEADERSHIP BRIEF Jeremy Kramer, jeremy.kramer.1@us.af.mil Air Mobility Command Test and Evaluation Squadron

OVERVIEW Motivation Program List/Overview Software Development Study SDLC Method Best Practices Questions

Motivation Office of the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation, DASDE (DT&E) Rotational Assignment, April-September 2018 Mr. Chris Deluca (Space, Cyber and Info Sys Director) Focus is to inform on Development Methods as directed by National Defense Authorization Act FY18 (sec. 873, 874) Produced Case Study White Paper with Leland Hewitt, Mark Price and Dr. Ferhan Kilical (MITRE) Hirsh Research Competition 2019 Coauthored with Lt Col Torrey J. Wagner working with Susan Gotta (AMCTES) and Dr. Kilical (MITRE)

Program List Joint Space Operations Center (JSpOC) Mission System (JMS), Air Force Distributed Common Ground System–Navy (DCGS-N) Increment 2, Navy Global Combat Support System–Joint (GCSS-J), Defense Information Systems Agency (DISA) Reserve Component Automation System (RCAS), Army Catapult/ANTS (Attack the Network Tool Suite), Joint Improvised-Threat Defeat Organization (JIDO)

Program Overview Program Description Group Service JMS Space command and control situational awareness System of Systems (SoS) Air Force DCGS-N Inc. 2 Tactical gateway for Navy-unique sensor data across the intelligence community SoS Navy GCSS-J Joint Military integrated logistics through applications and tools Web Apps DISA RCAS Integrated, web-based software solutions to manage mobilization, safety, personnel and force authorization requirements Army Catapult/ANTS Catapult integrates global intelligence data; ANTS addresses improvised explosive device threats JIDO

Software Development Agile Focus on small, frequent capability releases Values working software over comprehensive documentation Responds rapidly to changes in operations, technology and budgets Actively involving users throughout development ensuring high operational value Secure Development Operations (Secure DevOps) Strong use of automation (code, build, infrastructure/architecture, test) Continuous development, test, integration, and demonstration of software

Study Survey for PMO personnel completed between June and August 2018: Agile Development, specifically method, management tools and structure of teams Requirements tool usage Documents used during actual development Test planning, test events, test implementer Greatest successes and challenges and future goals Lessons learned and best practices in areas of: requirements detailed planning test events and execution

Software Development Life Cycle (SDLC) Method Program SDLC Tools Iteration Duration # of Iterations to a Release Sprint Teams JMS Scrum/ Waterfall Jira, in-house 1 month, 2 month integration 6 to 20 1 Integrator, 4-6 vendors DCGS-N Inc. 2 SAFe Jira 1 month 3 sprints per capability, 4 per Fleet release 6 working in parallel, started with 2 GCSS-J Scrum Rally, Jira 4, release capabilities as completed 4 in parallel RCAS SAFe, Secure DevOps Version-One 3 weeks 4 sprints, quarterly releases, 1 week release planning 18 teams across RCAS portfolio, 2 separate locations Catapult/ANTS Secure DevOps 2 days to 2 weeks 1 month minor, 3 months large N/A

Best Practices Develop a Tailored Approach with Continuous Delivery Web application suites more suitable for DevOps SoS complexity makes implementation difficult PMOs should consider tailored approach based on requirements and timeframes Continuous delivery provides smaller pieces of capability to the user community sooner

Best Practices Emphasize Obtaining Quality Feedback on Requirements and Solutions Active requirements development or gathering teams within the PMO refine requirements provided by the functional sponsor and/or users. Catapult/ANTS PMO has dedicated development engineers embedded at user facilities

Best Practices Invite Developers and Users to Participate in Government Testing Successful DT communicates defects back to the developer as quickly as possible. Facilitated developer participation aides this User involvement helps prepare system for OT JMS PMO provides access to a program instance at users’ home stations since obtaining time from dedicated operators for the test events can be difficult.

Best Practices Return on Investment for Automated Testing Depends on the Program Several PMOs recommended pursuing automated test tools, however both RCAS and Catapult/ANTS PMOs note that no single tool works best as “silver bullet.” RCAS and Catapult/ANTS have automated pipelines Catapult/ANTS pursues continuous risk management approach with SonarQube and Twistlock, to automatically verify 240/242 STIG mandated controls, and manually checks the remainder

Best Practices Transition to Agile is not a quick and easy process Secure DevOps programs developed processes over 3-5 year period Retraining of personnel, 30 percent change in skillsets 45th Test Squadron also investigating workforce retraining to better support automated testing or provide specific manual testing solutions DoD agencies (DAU, etc.) providing training for personnel as new guidance is released.

QUESTIONS Safely Deliver Timely, Accurate, and Objective Data and Analysis to AMC Decision Makers