1. Sarah A. Sheard Christopher L. Miller Software Productivity Consortium
The Shangri-La of ROI
Sarah A. Sheard
Christopher L. Miller
Software Productivity Consortium

2. Agenda There aren’t any hard numbers
We want to see Hard Numbers proving process improvement is a good investment
So now what?
There aren’t going to be any hard numbers
You wouldn’t be able to apply them
We’re in business to make money
Project justification required
Return must exceed costs
Process improvement is a project
If improving Systems Engineering processes increases our process capability, product quality, and saves money
… how much money will it save us?
No one would believe you if you did

3. There aren’t any hard numbers
Category
Range
Yearly cost of software process improvement (SPI) activities
$49K - $1,202K
Years engaged in SPI
1 – 9
Cost of SPI per software engineer
$490 - $2,004
Productivity gain per year
9% - 67%
Early detection gain per year (defects discovered pre-test)
6% - 25%
Yearly reduction in time to market
15% - 23%
Yearly reduction in post-release defects
10% - 94%
Value of investment in SPI (value returned on each $ invested)
Applicability
Time
Bias (Thought Experiment #1)

4. There aren’t going to be any hard numbers
Proprietary data
Productivity
Cycle Time
System development cost
Defect rates
Process improvement investment
Fidelity of estimates
Cost of SE activities per year
Rework avoided, Risks avoided, Better designs, Other
What is systems engineering anyway?
Twelve Roles '96
Three Types '00

5. If there were, you wouldn’t be able to use them$
What do the numbers mean?
Waste in current process?
Dependence on “smart people”
What is “productivity”?
Time-to-market return?
Procurement rules?
Separability of causes (Thought Experiment #2)

6. If you did, nobody would believe you anyway
No guarantees:
PI, done wrong, gives negative ROI
People believe what they want to believe
Franz paper isn’t “good enough”
Mobil Oil

7. Agenda Let’s review the points So now what? Making the SE PI case
Reasons for Investment
We’re in business to make money
Project justification required
Return must exceed costs
Process improvement is a project
If improving Systems Engineering processes increases our process capability, product quality, and saves money
… how much money will it save us?
Getting Started

8. There are ranges of numbers There aren’t any hard numbers
Category
Range
Yearly cost of software process improvement (SPI) activities
$49K - $1,202K
Years engaged in SPI
1 – 9
Cost of SPI per software engineer
$490 - $2,004
Productivity gain per year
9% - 67%
Early detection gain per year (defects discovered pre-test)
6% - 25%
Yearly reduction in time to market
15% - 23%
Yearly reduction in post-release defects
10% - 94%
Value of investment in SPI (value returned on each $ invested)
Can estimate value with these ranges
Identify similarities & adjust for differences
Document approach, rationale, and decisions
(just like any other estimation)
Applicability
Time
Bias (Thought Experiment #1)

9. There aren’t going to be any hard numbers
Consortium will collect and analyze the state
of the practice for determining ROI
PI pioneers will start to have processes,
products, and measures to determine ROI
Proprietary data
Productivity
Cycle Time
System development cost
Defect rates
Process improvement investment
Fidelity of estimates
Cost of SE activities per year
Rework avoided, Risks avoided, Better designs, Other
What is systems engineering anyway?

10. If there were, you wouldn’t be able to use them
If ROI numbers exist, use them appropriately:
Know what they represent
- Similar domain, development activities, …
Integrate assumptions into decisions
- Manage the uncertainty as a risk
What do the numbers mean?
Waste in current process?
Dependence on “smart people”
What is “productivity”?
Overtime, overhead, G&A
Time-to-market return?
Procurement rules?
Separability of causes (Thought Experiment #2) $$

11. If you did, nobody would believe you anyway
No guarantees:
PI, done wrong, gives negative ROI
People believe what they want to believe
Franz paper isn’t “good enough” ; Mobil Oil
Objective not to defend the ROI numbers,
- But to obtain support for systems
engineering process improvement
Use the ROI numbers as supporting information
- Avoid using it as the crux of your argument

12. Making the SE PI Case When no hard numbers exist, use following:
Establish goals and measures
Show ROI as goals are met
Use anecdotal evidence from technical journals
Use Peer Pressure - Competitor is Level 3
Value of Systems Engineering
Bottom line:
This makes sense.

13. Making the SE PI Case
Identify current implementations of SE not thought of as systems engineering activities
e.g.,contract deliverables
‘Systems Engineering’ may be a buzz word
If so, use another term
Program Management
e.g., Configuration management
Things aren’t getting done
e.g., large amount of rework and testing effort required

14. Reasons for Investment
External reasons
“Our customer complains”
Resource problems
“We can’t hire people fast enough”
Disconnects
“This change request is full of errors, yet everyone signed off on it! How can that be?”
Vicious cycles
“I wish we had found these problems before test. But who has time?”
These categories help identify the need for SE PI:
External reasons
“Our customer complains about problems with the delivered product. If we don’t improve, and fast, we’ll never win another contract from them.”
Resource problems
“In this job market we can’t hire people fast enough to staff all our programs. We have no choice but to become more efficient.”
Disconnects
“This change request is full of errors, yet everyone signed off on it! How can that be?”
Vicious cycles
“We find a lot of problems in test, when it’s really expensive to correct them. I sure wish we had found them earlier. But who has time?”

15. $ Getting Started What’s it worth to solve these problems?
Phrase problems as costs
“Two weeks of testing adds $50K.”
Solve the right problem
Keep problem from happening again
Proactive management avoids future problems
Get help!
Don’t waste time and money struggling
Start small
Learn from your successes and failures

16. Conclusions Management wants hard ROI numbers
But they do not currently exist
No common definition of ROI
Still need to make a case for systems engineering process improvement
Obtain buy-in on how to measure ROI
Start collecting data
Use these suggestions to get initial funding

17. Agenda Let’s review the points Let’s GO! So now what?
Making the SE PI case
Reasons for Investment
We’re in business to make money
Project justification required
Return must exceed costs
Process improvement is a project
If improving Systems Engineering processes increases our process capability, product quality, and saves money
… how much money will it save us?
Getting Started

18. Author Contact Information
Sarah Sheard (703)
Chris Miller (703)
Software Productivity Consortium
2214 Rock Hill Rd.
Herndon VA
Or see SPC booth in the exhibit area