1. The Software Dilemma
Ceci Albert

2. Copyright 2019 Carnegie Mellon University. All Rights Reserved
Copyright 2019 Carnegie Mellon University. All Rights Reserved. This material is based upon work funded and supported by the Department of Defense under Contract No. FA D-0002 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN "AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. [DISTRIBUTION STATEMENT A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution. This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at DM

3. Why You Care “Software is eating the world”
The system needs software to meet objectives
Software is everywhere in your system
Lifecycle costs are a big deal
Software will be continuously engineered, developed, integrated, tested, and deployed for a very long time
You need to comply with cyber requirements
“Software is eating the world”
—Marc Andreessen, WSJ, 2011
Correct latent errors
Adjust to changes in the environment
Address hardware obsolescence
Continuous innovation

4. Outline
Different Kinds of Software in Your System Clashing Development Lifecycles Some Acquisition Considerations Moving Forward…
Data Processing Software
Embedded Software
Test and Development Software
System Hardware Life Cycle
Processing Hardware Life Cycle
Software Life Cycle
Acquisition Strategy Considerations
RFP Considerations
Source Selection Considerations
Contract Execution Considerations

5. Different Kinds of Software in Your System
The Software Dilemma
Different Kinds of Software in Your System
This is a different set of skills and practices from other types of software development, like test software and command and control functions.
The three should be separated as a dependent set of activities.

6. Data Processing Software
What we are talking about:
Software that connects the system to the outside world
Software that plans and directs system operation
Software that processes business/mission data
This software uses commodity processing hardware and infrastructure that change frequently
There may be more than two categories of tactical software that will be acquired in this program, but embedded and command are the two that came up in our conversation.
This is an arena well suited to DevOps/SAFe continuous engineering software design and fielding practices. The computing hardware and networking technologies in this world are commodities and will change frequently (much more so than in the embedded case). As such there should be a different support environment and acquisition cadence.
Virtualization, containerization and open standards will insulate the care and feeding of the product from the vagaries of the commercial hardware and support software environment and should be used extensively. The Open Systems Architecture Contract Guidebook for Program Managers might be helpful.

7. Embedded Software What we are talking about:
The software that controls the behavior of or processes data from the system hardware, especially unique elements in weapon systems that also need to address anti-tamper requirements
Performance critical
Mission critical
Safety critical
Software development is closely tied to the hardware component development
Required for proof of concept
Coded as part of proof of concept before component is built or processors are selected. OR
Requirements dependent on component design.
The use of real-time middlewares and consortium technical frameworks can help insulate the PMO from the hardware/software dependence that naturally occurs in embedded products. By the time the hardware component is defined sufficiently, it is too late to develop the software

8. Test and Development Software
What we are talking about:
Automated tools that support the software development pipeline
Software that systematically looks for common coding errors
Software that systematically measures software performance
Tools are closely linked to the developer’s culture and processes but will be needed for sustainment of the software
What types of software are in your system?
Software may be eating the world, but software tooling is eating software – and you need to understand the tools your developer is using … they will have to be used for the life of the system.”
s such, the program office needs to capture information on how the tooling is selected, configured and used. The program office should ensure it preserves future acquisition decisions to move sustainment to a different contractor or government lab. Should they ever decide to do this, capturing information on the tooling will de-risk those future alternatives.
You will spend as much on test software as you will on the tactical code. Treat it as a first-class citizen. The schedule for the overall project should show how the test software matches up to the operational software needs for testing (with notable differences between embedded and Planning/Command software testing needs)
Pay special attention to the use of automated testing. You should encourage its use, but make sure you get transparency on how it is acquired, configured, used, and delivered.
Test software needs to be delivered, complete with documentation so that the program can have down-stream life cycle alternative resources perform capability integration, cyber-vulnerability patching, hardware obsolescence impacts to software, and other maintenance functions.

9. Clashing Development Lifecycles
The Software Dilemma
Clashing Development Lifecycles

10. System Hardware Development Lifecycle
What we are talking about:
Hardware that has to be developed for your system
Vehicles: ship, air frame, rocket body
Components: engines, sensors
Lifecycle patterns:
Milestone decision points
System meets requirements for reasonable cost and risk
System design meets requirements at reasonable cost and risk
System built is suitable for production in large numbers
System is ready for operational deployment
Waterfall development works
Requirements, design, build, integrate, test, deploy

11. Processing Hardware Lifecycle
What we are talking about:
Processors that will operate, test, and maintain your system
On-board processors
Mission support processors
Cloud
The software infrastructure: operating systems, database and network services
Lifecycle patterns:
Decision points
Form, fit, and function defined at system PDR/CDR
“Buy” decision delayed until as late as possible to maximize value
COTS process works best
Buy the highest performing processor available
Select a compatible software infrastructure stack with processor
Today, these are almost always COTS
Field-Programmable Gate Array (FPGA)
Graphics Processing Unit (GPU)
Central Processing Unit (CPU)
Maximize value = the later you wait the more capable the processor or the lower the cost for the same performance point
= avoid obsolescence at system IOC
Computing Processing Units (CPU)

12. Software Lifecycle What we are talking about:
Software that has to be developed for your system
Off-the-shelf software that was developed for someone else
Lifecycle patterns:
Decision points
Similar to system hardware, but they don’t line up!
Incremental development works best
Concurrent requirements, design, build, integrate, test, deploy
Embedded
Test
Planning / Command and Control

13. System Design
SRR
PDR
CDR
System
PDR
CDR
Component Software
Processor
Embedded Software
Embedded Software
Test and Development Software?
How/when will you get visibility into your different software types?

14. Some Acquisition Considerations
The Software Dilemma
Some Acquisition Considerations

15. Acquisition Strategy Considerations
Where is the software in your system? What types?
Who are your stakeholder organizations, and how will they flow down dependent requirements?
Who has the authority to halt progress, and what value do they add?
Does the PMO have the necessary software expertise?
Do you need to contract for software expertise?
Do the potential offerors have the necessary software expertise?
How will they execute the system’s different software domains?
What visibility will the PMO have into software development?
How will the software be sustained for the next 50 years?
Plan for continuous engineering, development, and deployment
It is not just about buying the software; it about feeding the beast

16. RFP Considerations PWS Are tasks outcome based? QASP
For example, delivery of a minimally viable product in 6 months with new features every 30 days thereafter
QASP
How will performance be measured?
Has legacy performance been benchmarked?
CDRLs
Do you have all/only the data you need?
Do you have a plan for review of the data?
How will you verify the data deliveries?
How does each data item align with decisions that will need to be made?

17. Source Selection Considerations
Have the offeror write a story about
how software will function in the operational environment (understanding of need)
how offeror will interact with the PMO in managing this contract
how offeror will interact with other stakeholders to validate software changes
how offeror will build, test, and deploy the software in the program
how the software will be sustained for the next 50 years
How will you know the offeror is competent? How will you distinguish between two competent offerors?
Show how the different kinds of software are going to be managed. How are they different, how are they ultimately integrated?
Show what software components are encapsulated so they can proceed independently and a problem can be isolated for correction
Evaluate make/buy decisions. What is the provenance for the reuse software?
Has the offeror demonstrated competence: consistent development tempo; expertise in the key technical areas?
Are they setting up your environment for continuous change?
How will they build/test/deploy the software in the program (Technical Approach)
Is the offeror able to adequately describe the work that has to be done for the system software to operate effectively, securely?
Show how the different kinds of software are going to be managed.  How are they different, how are they ultimately integrated?
Evaluate make/buy decisions.  What is the provenance for the reuse software? Selection criteria for of COTS and Open Source
How will software problems be reported, tracked and resolved?
Are the technical performance metrics meaningful? Timely?
Is the offeror able to adequately describe the functions that the software must support?
Is the offeror able to adequately describe the relationships between the software, the processing hardware, and the  other system components?
How to they describe management of software (Management Approach):
Does the offeror have management process and procedures expected to ensure successful software development and deployment?
What visibility is the PMO going to get into project?
Will the offeror have valid, tested management processes?
Are the management metrics meaningful?  Timely?
Deliverables and associated Data Rights strategy
How will the software will be developed, tested, and deployed (Technical Capability)
Does the offeror have, or can the offeror be reasonably expected to acquire, the technical skills and knowledge needed?
Has the contractor demonstrated competence: consistent development tempo; expertise in the key technical areas?
How do they expect to interact with the PMO (Risk)
What visibility is the PMO going to get into software development, integration, test, deployment?
Who is the “Product Owner” (requirements manager/prioritizer)
What decisions will the program office be involved in?  (time for introspection: does the PMO have the expertise to make these decisions?)
What automated tools are planned to be used in the development, test environments?  What experience in using these tools has the offeror demonstrated in similar programs?

18. Contract Execution Considerations
How does the PMO interact with the contractor? With the subcontractors? What metrics will the contractor use to track software development progress? What metrics will the contractor use to track software system performance?
Work out how the PMO will participate in design trades

19. The Software Dilemma
Moving Forward

20. Moving Forward Beware of silver bullets!
Software-controlled everything has added a new level of complexity to system acquisition without taking any of the old challenges away One size does NOT fit all – and your system may have many types of software We no longer have a software crisis, we have an ongoing dilemma
Beware of silver bullets!