The Software Dilemma Ceci Albert

Copyright 2019 Carnegie Mellon University. All Rights Reserved Copyright 2019 Carnegie Mellon University. All Rights Reserved. This material is based upon work funded and supported by the Department of Defense under Contract No. FA8702-15-D-0002 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. [DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu. DM19-0138

Why You Care “Software is eating the world” The system needs software to meet objectives Software is everywhere in your system Lifecycle costs are a big deal Software will be continuously engineered, developed, integrated, tested, and deployed for a very long time You need to comply with cyber requirements “Software is eating the world” —Marc Andreessen, WSJ, 2011 Correct latent errors Adjust to changes in the environment Address hardware obsolescence Continuous innovation

Outline Different Kinds of Software in Your System Clashing Development Lifecycles Some Acquisition Considerations Moving Forward… Data Processing Software Embedded Software Test and Development Software System Hardware Life Cycle Processing Hardware Life Cycle Software Life Cycle Acquisition Strategy Considerations RFP Considerations Source Selection Considerations Contract Execution Considerations

Different Kinds of Software in Your System The Software Dilemma Different Kinds of Software in Your System This is a different set of skills and practices from other types of software development, like test software and command and control functions. The three should be separated as a dependent set of activities.

Data Processing Software What we are talking about: Software that connects the system to the outside world Software that plans and directs system operation Software that processes business/mission data This software uses commodity processing hardware and infrastructure that change frequently There may be more than two categories of tactical software that will be acquired in this program, but embedded and command are the two that came up in our conversation. This is an arena well suited to DevOps/SAFe continuous engineering software design and fielding practices. The computing hardware and networking technologies in this world are commodities and will change frequently (much more so than in the embedded case). As such there should be a different support environment and acquisition cadence. Virtualization, containerization and open standards will insulate the care and feeding of the product from the vagaries of the commercial hardware and support software environment and should be used extensively. The Open Systems Architecture Contract Guidebook for Program Managers might be helpful.

Embedded Software What we are talking about: The software that controls the behavior of or processes data from the system hardware, especially unique elements in weapon systems that also need to address anti-tamper requirements Performance critical Mission critical Safety critical Software development is closely tied to the hardware component development Required for proof of concept Coded as part of proof of concept before component is built or processors are selected. OR Requirements dependent on component design. The use of real-time middlewares and consortium technical frameworks can help insulate the PMO from the hardware/software dependence that naturally occurs in embedded products. By the time the hardware component is defined sufficiently, it is too late to develop the software

Test and Development Software What we are talking about: Automated tools that support the software development pipeline Software that systematically looks for common coding errors Software that systematically measures software performance Tools are closely linked to the developer’s culture and processes but will be needed for sustainment of the software What types of software are in your system? Software may be eating the world, but software tooling is eating software – and you need to understand the tools your developer is using … they will have to be used for the life of the system.” s such, the program office needs to capture information on how the tooling is selected, configured and used. The program office should ensure it preserves future acquisition decisions to move sustainment to a different contractor or government lab. Should they ever decide to do this, capturing information on the tooling will de-risk those future alternatives. You will spend as much on test software as you will on the tactical code. Treat it as a first-class citizen. The schedule for the overall project should show how the test software matches up to the operational software needs for testing (with notable differences between embedded and Planning/Command software testing needs) Pay special attention to the use of automated testing. You should encourage its use, but make sure you get transparency on how it is acquired, configured, used, and delivered. Test software needs to be delivered, complete with documentation so that the program can have down-stream life cycle alternative resources perform capability integration, cyber-vulnerability patching, hardware obsolescence impacts to software, and other maintenance functions.

Clashing Development Lifecycles The Software Dilemma Clashing Development Lifecycles

System Hardware Development Lifecycle What we are talking about: Hardware that has to be developed for your system Vehicles: ship, air frame, rocket body Components: engines, sensors Lifecycle patterns: Milestone decision points System meets requirements for reasonable cost and risk System design meets requirements at reasonable cost and risk System built is suitable for production in large numbers System is ready for operational deployment Waterfall development works Requirements, design, build, integrate, test, deploy

Processing Hardware Lifecycle What we are talking about: Processors that will operate, test, and maintain your system On-board processors Mission support processors Cloud The software infrastructure: operating systems, database and network services Lifecycle patterns: Decision points Form, fit, and function defined at system PDR/CDR “Buy” decision delayed until as late as possible to maximize value COTS process works best Buy the highest performing processor available Select a compatible software infrastructure stack with processor Today, these are almost always COTS Field-Programmable Gate Array (FPGA) Graphics Processing Unit (GPU) Central Processing Unit (CPU) Maximize value = the later you wait the more capable the processor or the lower the cost for the same performance point = avoid obsolescence at system IOC Computing Processing Units (CPU)

Software Lifecycle What we are talking about: Software that has to be developed for your system Off-the-shelf software that was developed for someone else Lifecycle patterns: Decision points Similar to system hardware, but they don’t line up! Incremental development works best Concurrent requirements, design, build, integrate, test, deploy Embedded Test Planning / Command and Control

System Design SRR PDR CDR System PDR CDR Component Software Processor Embedded Software Embedded Software Test and Development Software? How/when will you get visibility into your different software types?

Some Acquisition Considerations The Software Dilemma Some Acquisition Considerations

Acquisition Strategy Considerations Where is the software in your system? What types? Who are your stakeholder organizations, and how will they flow down dependent requirements? Who has the authority to halt progress, and what value do they add? Does the PMO have the necessary software expertise? Do you need to contract for software expertise? Do the potential offerors have the necessary software expertise? How will they execute the system’s different software domains? What visibility will the PMO have into software development? How will the software be sustained for the next 50 years? Plan for continuous engineering, development, and deployment It is not just about buying the software; it about feeding the beast

RFP Considerations PWS Are tasks outcome based? QASP For example, delivery of a minimally viable product in 6 months with new features every 30 days thereafter QASP How will performance be measured? Has legacy performance been benchmarked? CDRLs Do you have all/only the data you need? Do you have a plan for review of the data? How will you verify the data deliveries? How does each data item align with decisions that will need to be made?

Source Selection Considerations Have the offeror write a story about how software will function in the operational environment (understanding of need) how offeror will interact with the PMO in managing this contract how offeror will interact with other stakeholders to validate software changes how offeror will build, test, and deploy the software in the program how the software will be sustained for the next 50 years How will you know the offeror is competent? How will you distinguish between two competent offerors? Show how the different kinds of software are going to be managed. How are they different, how are they ultimately integrated? Show what software components are encapsulated so they can proceed independently and a problem can be isolated for correction Evaluate make/buy decisions. What is the provenance for the reuse software? Has the offeror demonstrated competence: consistent development tempo; expertise in the key technical areas? Are they setting up your environment for continuous change? How will they build/test/deploy the software in the program (Technical Approach) Is the offeror able to adequately describe the work that has to be done for the system software to operate effectively, securely? Show how the different kinds of software are going to be managed.  How are they different, how are they ultimately integrated? Evaluate make/buy decisions.  What is the provenance for the reuse software? Selection criteria for of COTS and Open Source How will software problems be reported, tracked and resolved? Are the technical performance metrics meaningful? Timely? Is the offeror able to adequately describe the functions that the software must support? Is the offeror able to adequately describe the relationships between the software, the processing hardware, and the  other system components? How to they describe management of software (Management Approach): Does the offeror have management process and procedures expected to ensure successful software development and deployment? What visibility is the PMO going to get into project? Will the offeror have valid, tested management processes? Are the management metrics meaningful?  Timely? Deliverables and associated Data Rights strategy How will the software will be developed, tested, and deployed (Technical Capability) Does the offeror have, or can the offeror be reasonably expected to acquire, the technical skills and knowledge needed? Has the contractor demonstrated competence: consistent development tempo; expertise in the key technical areas? How do they expect to interact with the PMO (Risk) What visibility is the PMO going to get into software development, integration, test, deployment? Who is the “Product Owner” (requirements manager/prioritizer) What decisions will the program office be involved in?  (time for introspection: does the PMO have the expertise to make these decisions?) What automated tools are planned to be used in the development, test environments?  What experience in using these tools has the offeror demonstrated in similar programs?

Contract Execution Considerations How does the PMO interact with the contractor? With the subcontractors? What metrics will the contractor use to track software development progress? What metrics will the contractor use to track software system performance? Work out how the PMO will participate in design trades

The Software Dilemma Moving Forward

Moving Forward Beware of silver bullets! Software-controlled everything has added a new level of complexity to system acquisition without taking any of the old challenges away One size does NOT fit all – and your system may have many types of software We no longer have a software crisis, we have an ongoing dilemma Beware of silver bullets!