1. Developmental Test and Requirements: Best Practices of Successful Information Systems using Agile Methods
LEADERSHIP BRIEF
Jeremy Kramer, Air Mobility Command Test and Evaluation Squadron

2. OVERVIEW Motivation Program List/Overview Software Development Study
SDLC Method
Best Practices
Questions

3. Motivation
Office of the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation, DASDE (DT&E)
Rotational Assignment, April-September 2018
Mr. Chris Deluca (Space, Cyber and Info Sys Director)
Focus is to inform on Development Methods as directed by National Defense Authorization Act FY18 (sec. 873, 874)
Produced Case Study White Paper with Leland Hewitt, Mark Price and Dr. Ferhan Kilical (MITRE)
Hirsh Research Competition 2019
Coauthored with Lt Col Torrey J. Wagner
working with Susan Gotta (AMCTES) and Dr. Kilical (MITRE)

4. Program List
Joint Space Operations Center (JSpOC) Mission System (JMS), Air Force
Distributed Common Ground System–Navy (DCGS-N) Increment 2, Navy
Global Combat Support System–Joint (GCSS-J), Defense Information Systems Agency (DISA)
Reserve Component Automation System (RCAS), Army
Catapult/ANTS (Attack the Network Tool Suite), Joint Improvised-Threat Defeat Organization (JIDO)

5. Program Overview Program Description Group Service JMS
Space command and control situational awareness
System of Systems (SoS)
Air Force
DCGS-N Inc. 2
Tactical gateway for Navy-unique sensor data across the intelligence community
SoS
Navy
GCSS-J
Joint Military integrated logistics through applications and tools
Web Apps
DISA
RCAS
Integrated, web-based software solutions to manage mobilization, safety, personnel and force authorization requirements
Army
Catapult/ANTS
Catapult integrates global intelligence data; ANTS addresses improvised explosive device threats
JIDO

6. Software Development Agile
Focus on small, frequent capability releases
Values working software over comprehensive documentation
Responds rapidly to changes in operations, technology and budgets
Actively involving users throughout development ensuring high operational value
Secure Development Operations (Secure DevOps)
Strong use of automation (code, build, infrastructure/architecture, test)
Continuous development, test, integration, and demonstration of software

7. Study Survey for PMO personnel completed between June and August 2018:
Agile Development, specifically method, management tools and structure of teams
Requirements tool usage
Documents used during actual development
Test planning, test events, test implementer
Greatest successes and challenges and future goals
Lessons learned and best practices in areas of:
requirements
detailed planning
test events and execution

8. Software Development Life Cycle (SDLC) Method
Program
SDLC
Tools
Iteration Duration
# of Iterations to a Release
Sprint Teams
JMS
Scrum/ Waterfall
Jira, in-house
1 month, 2 month integration
6 to 20
1 Integrator, 4-6 vendors
DCGS-N Inc. 2
SAFe
Jira
1 month
3 sprints per capability, 4 per Fleet release
6 working in parallel, started with 2
GCSS-J
Scrum
Rally, Jira
4, release capabilities as completed
4 in parallel
RCAS
SAFe, Secure DevOps
Version-One
3 weeks
4 sprints, quarterly releases, 1 week release planning
18 teams across RCAS portfolio, 2 separate locations
Catapult/ANTS
Secure DevOps
2 days to 2 weeks
1 month minor, 3 months large
N/A

9. Best Practices Develop a Tailored Approach with Continuous Delivery
Web application suites more suitable for DevOps
SoS complexity makes implementation difficult
PMOs should consider tailored approach based on requirements and timeframes
Continuous delivery provides smaller pieces of capability to the user community sooner

10. Best Practices
Emphasize Obtaining Quality Feedback on Requirements and Solutions
Active requirements development or gathering teams within the PMO refine requirements provided by the functional sponsor and/or users.
Catapult/ANTS PMO has dedicated development engineers embedded at user facilities

11. Best Practices
Invite Developers and Users to Participate in Government Testing
Successful DT communicates defects back to the developer as quickly as possible.
Facilitated developer participation aides this
User involvement helps prepare system for OT
JMS PMO provides access to a program instance at users’ home stations since obtaining time from dedicated operators for the test events can be difficult.

12. Best Practices
Return on Investment for Automated Testing Depends on the Program
Several PMOs recommended pursuing automated test tools, however both RCAS and Catapult/ANTS PMOs note that no single tool works best as “silver bullet.”
RCAS and Catapult/ANTS have automated pipelines
Catapult/ANTS pursues continuous risk management approach with SonarQube and Twistlock, to automatically verify 240/242 STIG mandated controls, and manually checks the remainder

13. Best Practices Transition to Agile is not a quick and easy process
Secure DevOps programs developed processes over 3-5 year period
Retraining of personnel, 30 percent change in skillsets
45th Test Squadron also investigating workforce retraining to better support automated testing or provide specific manual testing solutions
DoD agencies (DAU, etc.) providing training for personnel as new guidance is released.

14. QUESTIONS Safely Deliver Timely, Accurate, and Objective
Data and Analysis to AMC Decision Makers