1. Software Risk Management and the COCOMO Model
Barry Boehm
USC
Fall 2011

2. Outline What is Software Risk Management? When should you do it?
Continuously from Day One
How should you do it?
Risk Assessment
Cost risk assessment and COCOMO II
Risk Control
Conclusions
4/17/2017

3. What is Software Risk Management?
An approach for early identification and mitigation of critical project uncertainties
Learning early and cheaply
Narrowing the Cone of Uncertainty
Avoiding expensive late rework
4/17/2017

4. Risk of Delaying Risk Management: Software10 20 50
100
200
500
1000
Relative cost to fix defect 2 1 5
Requirements Design Code Development Acceptance Operation
test test
Smaller software projects
Larger software projects
Median (TRW survey)
80%
20%
SAFEGUARD
GTE
IBM-SSD
Phase in Which defect was fixed
4/17/2017

5. Steeper Cost-to-fix for High-Risk Elements
TRW Project B
1005 SPR’s
100 90 80
TRW Project A
373 SPR’s 70
% of
Cost to
Fix
SPR’s 60 50
Major Rework Sources:
Off-Nominal Architecture-Breakers
A - Network Failover
B - Extra-Long Messages 40 30 20 10 10 20 30 40 50 60 70 80 90
100
% of Software Problem Reports (SPR’s)
4/17/2017

6. Risk of Delaying Risk Management: Systems
Blanchard- Fabrycky, 1998
(Cone of Uncertainty)

7. The Fundamental Risk Management Metric
Risk Exposure RE = Prob (Loss) * Size (Loss)
“Loss” – of stakeholders’ value
financial; reputation; quality of service, …
For multiple sources of loss:
RE =  [Prob (Loss) * Size (Loss)]source
sources
4/17/2017

8. Prioritizing Risks: Risk Exposure Risk Exposure = (Probability) (Loss of Utility)
High
Check
Utility -
Loss Estimate
Major
Risk
Risk Probability
Check
Probability
Estimate
Little
Risk
Low
Loss of Utility
Low
High
4/17/2017

9. Risk Exposure Factors (Satellite Experiment Software)
Unsatisfactory Outcome (UO)
Prob (UO)
Loss (UO) 10 8 7 9 3 4 1 5 2
Risk Exposure
3 - 5
4 - 8 5 6 8 1 2 45 15 24 8 30 7 4
A. S/ W error kills experiment
B. S/ W error loses key data
C. Fault tolerance features cause unacceptable
performance
D. Monitoring software reports unsafe condition
as safe
E. Monitoring software reports safe condition
as unsafe
F. Hardware delay causes schedule overrun
G. Data reduction software errors cause extra
work
H. Poor user interface causes inefficient
operation
I. Processor memory insufficient
J. DBMS software loses derived data
4/17/2017

10. Risk Exposure Factors and Contours: Satellite Experiment Software
4/17/2017

11. Risk Reduction Leverage (RRL) Equivalent to Return on Investment
BEFORE -
AFTER
RRL =
RISK REDUCTION COST
Spacecraft Example
LONG DURATION
TEST
FAILURE MODE
TESTS
LOSS (UO)
PROB (UO) RE
$20M
0.2
$4M
$20M
0.2
$4M B B
PROB (UO) RE
0.05
$1M
0.07
$1.4M A A
COST
$2M
$0.26M
4-1
4- 1.4
= 1.5
= 10
RRL 2
0.26
4/17/2017

12. Outline What is Software Risk Management? When should you do it?
Continuously from Day One
How should you do it?
Risk Assessment
Cost risk assessment and COCOMO II
Risk Control
Conclusions
4/17/2017

13. Early Risk Resolution Quotes
“In architecting a new software program, all the serious mistakes are made on the first day.”
Robert Spinrad, VP-Xerox, 1988
“If you don’t actively attack the risks, the risks will actively attack you.”
Tom Gilb, 1988
4/17/2017

14. Day One Temptations to Avoid
It’s too early to think about risks. We need to:
Finalize the requirements
Maximize our piece of the pie
Converge on the risk management organization, forms, tools, and procedures. Don’t put the cart before the horse.
The real horse is the risks, and it’s leaving the barn
Don’t sit around laying out the seats on the cart
while this happens. Work this concurrently.
4/17/2017

15. Waterfall Model Risk Profile
Requirements
analysis
The most critical risks are architectural:
Performance (size, time, accuracy)
Interface integrity
System qualities (adaptability,
portability, etc.)
Specifications &
models
Design R i s k
Specifications & models
Code and unit test
Tested code
Subsystem integration
Executable subsystem
System integration
Executable system
Time
4/17/2017

16. Architectural prototype Draft specifications & models
ICM Risk Profile
Exploration
Architectural prototype
Draft specifications & models
Foundations
Initial executable system
Refined specifications & models R i s k
Development
Iteration 3...
Intermediate system
Operations
Final system
Time
4/17/2017

17. The ICM and Continuous Risk Management
Anchor Point Milestones
Synchronize, stabilize concurrency via FEDs
The Incremental Commitment Life Cycle Process: Overview
This slide shows how the ICM spans the life cycle process from concept exploration to operations. Each phase culminates with an anchor point milestone review. At each anchor point, there are 4 options, based on the assessed risk of the proposed system. Some options involve go-backs. These options result in many possible process paths.
The life cycle is divided into two stages: Stage I of the ICM (Definition) has 3 decision nodes with 4 options/node, culminating with incremental development in Stage II (Development and Operations). Stage II has an additional 2 decision nodes, again with 4 options/node.
One can use ICM risk patterns to generate frequently-used processes with confidence that they fit the situation. Initial risk patterns can generally be determined in the Exploration phase. One then proceeds with development as a proposed plan with risk-based evidence at the VCR milestone, adjusting in later phases as necessary. For complex systems, a result of the Exploration phase would be the Prototyping and Competition Plan discussed above.
Risks associated with the system drive the life cycle process. Information about the risk(s) (feasibility assessments) supports the decision to proceed, adjust scope or priorities, or cancel the program.
Risk patterns determine life cycle process
4/17/2017 17

18. Nature of FEDs and Anchor Point Milestones
Evidence provided by developer and validated by independent experts that:
If the system is built to the specified architecture, it will
Satisfy the specified operational concept and requirements
Capability, interfaces, level of service, and evolution
Be buildable within the budgets and schedules in the plan
Generate a viable return on investment
Generate satisfactory outcomes for all of the success-critical stakeholders
Shortfalls in evidence are uncertainties and risks
Should be resolved or covered by risk management plans
Assessed in increasing detail at major anchor point milestones
Serves as basis for stakeholders’ commitment to proceed
Serves to synchronize and stabilize concurrently engineered elements
Nature of FEDs and Anchor Point Milestones
The Feasibility Evidence Description (FED) does not assess a single sequentially developed system definition element, but the consistency, compatibility, and feasibility of several concurrently-engineered elements. To make this concurrency work, a set of anchor point milestone reviews are performed to ensure that the many concurrent activities are synchronized, stabilized, and risk-assessed at the end of each phase. Each of these anchor point milestone reviews is focused on developer-produced and expert-validated evidence, documented in the FED, to help the key stakeholders determine whether to proceed into the next level of commitment.
The FED is based on evidence from simulations, models, or experiments with planned technologies and increasingly detailed analysis of development approaches and projected productivity rates. The parameters used in the analyses should be based on measured component performance or on historical data showing relevant past performance, cost estimation accuracy, and actual developer productivity rates.
A shortfall in feasibility evidence indicates a level of program execution uncertainty and a source of program risk. It is often not possible to fully resolve all risks at a given point in the development cycle, but known, unresolved risks need to be identified and covered by risk management plans, including the necessary staffing and funding to address them. The nature of the evidence shortfalls, the strength and affordability of the risk management plans, and the stakeholders’ degrees of risk acceptance or avoidance will determine their willingness to commit the necessary resources to proceed. A program with risks is not necessarily bad, particularly if it has strong risk management plans. A program with no risks may be high on achievability, but low on ability to produce a timely competitive advantage.
A program more familiar with a sequential waterfall or V-model can achieve most of the effect of a FED-based anchor point milestone review by adding a FED to the set of artifacts to be developed and reviewed for adequacy and satisfaction at a System requirements Review (SRR), System Functional Review (SFR), or Preliminary Design Review (PDR). In principle, some guidance documents indicate that such feasibility evidence should be produced and reviewed. But since the evidence is not generally specified as a developer deliverable and is vaguely defined, it is generally inadequate as a basis for stakeholder commitments.
Can be used to strengthen current schedule- or event-based reviews
4/17/2017
File: Feasibility Evidence Developmentv8

19. Problems Encountered without FED: 15-Month Architecture Rework Delay
Required Architecture:
Custom; many cache processors
Original Architecture:
Modified
Client-Server 1 2 3 4 5
Response Time (sec)
Original Spec
After Prototyping
Original Cost
Problems Encountered without FED
In the early 1980s, a large government organization contracted with TRW to develop an ambitious information query and analysis system. The system would provide more than 1,000 users, spread across a large building complex, with powerful query and analysis capabilities for a large and dynamic database.
TRW and the customer specified the system using a classic sequential-engineering waterfall development model. Based largely on user need surveys, an oversimplified high-level performance analysis, and a short deadline for getting the TBDs out of the requirements specification, they fixed into the contract a requirement for a system response time of less than one second.
Subsequently, the software architects found that subsecond performance could only be provided via a highly customized design that attempted to anticipate query patterns and cache copies of data so that each user’s likely data would be within one second’s reach (a 1980’s precursor of Google). The resulting hardware architecture had more than 25 super-midicomputers busy caching data according to algorithms whose actual performance defied easy analysis. The scope and complexity of the hardware-software architecture brought the estimated cost of the system to nearly $100 million, driven primarily by the requirement for a one-second response time.
Faced with this unattractive prospect (far more than the customer’s budget for the system), the customer and developer decided to develop a prototype of the system’s user interface and representative capabilities to test. The results showed that a four-second response time would satisfy users 90 percent of the time. A four-second response time, with special handling for high-priority transactions, dropped development costs closer to $30 million. Thus, the premature specification of a 1-second response time neglected the risk of creating an overexpensive and time-consuming system development. Fortunately, in this case, the only loss was the wasted effort on the expensive-system architecture and a 15 month delay in delivery. More frequently, such rework is done only after the expensive full system is delivered and found still too slow and too expensive to operate.
4/17/2017
File: Feasibility Evidence Developmentv8

20. Problems Avoidable with FED
Attempt to validate 1-second response time
Commercial system benchmarking and architecture analysis: needs expensive custom solution
Prototype: 4-second response time OK 90% of the time
Negotiate response time ranges
2 seconds desirable
4 seconds acceptable with some 2-second special cases
Benchmark commercial system add-ons to validate their feasibility
Present solution and feasibility evidence at anchor point milestone review
Result: Acceptable solution with minimal delay
Problems Avoidable with FED Had the developers been required to deliver a FED showing evidence of feasibility of the one-second response time, they would have run benchmarks on the best available commercial query systems, using representative user workloads, and would have found that the best that they could do was about a 2.5 second response time, even with some preprocessing to reduce query latency. They would have performed a top-level architecture analysis of custom solutions, and concluded that such solutions were in the $100M cost range. They would have shared these results with the customer in advance of any key reviews, and found that the customer would prefer to explore the feasibility of a system with a commercially-supportable response time. They would have done user interface prototyping and found that 4 second response time was acceptable 90% of the time much earlier.
As some uncertainties still existed about the ability to address the remaining 10% of the queries, the customer and developer would have agreed to avoid repeating the risky specification of a fixed response time requirement, and instead to define a range of desirable-to-acceptable response times, with an award fee provided for faster performance. They would also have agreed to reschedule the next milestone review to give the developer time and budget to present evidence of the most feasible solution available, using the savings over the prospect of a $100M system development as rationale. This would have put the project on a more solid success track over a year before the actual project discovered and rebaselined itself, and without the significant expense that went into the unaffordable architecture definition.
4/17/2017
File: Feasibility Evidence Developmentv8

21. Reducing Software Cost-to-Fix: CCPDS-R - Royce, 1998
Architecture first
-Integration during the design phase
-Demonstration-based evaluation
Risk Management
Configuration baseline change metrics:
RATIONAL
S o f t w a r e C o r p o r a t I o n
Project Development Schedule 15 20 25 30 35 40 10
Design
Changes
Implementation Changes
Maintenance
Changes and ECP’s
Hours
Change
4/17/2017

22. Outline What is Software Risk Management? When should you do it?
Continuously from Day One
How should you do it?
Risk Assessment
Cost risk assessment and COCOMO II
Risk Control
Conclusions
4/17/2017

23. Software Risk Management
Identification
Checklists
Decision driver analysis
Assumption analysis
Decomposition
Performance models
Cost models
Network analysis
Decision analysis
Quality factor analysis
Risk exposure
Risk leverage
Compound risk reduction
Buying information
Risk avoidance
Risk transfer
Risk reduction
Risk element planning
Risk plan integration
Prototypes
Simulations
Benchmarks
Analyses
Staffing
Milestone tracking
Top-10 tracking
Risk reassessment
Corrective action
Risk
Assessment
Risk
Analysis
Risk
Prioritization
Risk
Management
Risk mgmt
Planning
Risk
Control
Risk
Resolution
Risk
Monitoring
4/17/2017

24. Risk Identification Techniques
Risk-item checklists
Decision driver analysis
Comparison with experience
Win-lose, lose-lose situations
Decomposition
Pareto 80 – 20 phenomena
Task dependencies
Murphy’s law
Uncertainty areas
Model Clashes
4/17/2017

25. Top 10 Risk Items: 1989 and 1995 1995 1989 Personnel shortfalls
Schedules and budgets
Wrong software functions
Wrong user interface
Gold plating
Requirements changes
Externally-furnished components
Externally-performed tasks
Real-time performance
Straining computer science
Personnel shortfalls
Schedules, budgets, process
COTS, external components
Requirements mismatch
User interface mismatch
Architecture, performance, quality
Requirements changes
Legacy software
Externally-performed tasks
Straining computer science
4/17/2017

26. Example Risk-item Checklist: Staffing
Will you project really get all the best people?
Are there critical skills for which nobody is identified?
Are there pressures to staff with available warm bodies?
Are there pressures to overstaff in the early phases?
Are the key project people compatible?
Do they have realistic expectations about their project job?
Do their strengths match their assignment?
Are they committed full-time?
Are their task prerequisites (training, clearances, etc.) satisfied?
4/17/2017

27. The Top Ten Software Risk Items
Risk Management Techniques
1. Personnel Shortfalls
Staffing with top talent; key personnel
agreements; incentives; team-building; training;
tailoring process to skill mix; peer reviews
2. Unrealistic schedules
and budgets
Business case analysis; design to cost; incremental
development; software reuse; requirements descoping;
adding more budget and schedule
3. COTS; external components
Qualification testing; benchmarking; prototyping; reference checking; compatibility analysis; vendor analysis; evolution support analysis
4. Requirements mismatch;
gold plating
Stakeholder win-win negotiation; business case
analysis; mission analysis; ops-concept formulation;
user surveys; prototyping; early users’ manual;
design/develop to cost
5. User interface mismatch
Prototyping; scenarios; user characterization
(functionality, style, workload)
4/17/2017

28. The Top Ten Software Risk Items (Concluded)
6. Architecture, performance,
quality
Architecture tradeoff analysis and review boards;
simulation; benchmarking; modeling; prototyping;
instrumentation; tuning
7. Requirements changes
High change threshold; information
hiding; incremental development (defer
changes to later increments)
8. Legacy software
Design recovery; phaseout options analysis;
wrappers/mediators; restructuring
9. Externally-performed
tasks
Reference checking; pre-award audits;
award-fee contracts; competitive design
or prototyping; team-building
10. Straining Computer
Science capabilities
Technical analysis; cost-benefit analysis;
prototyping; reference checking
4/17/2017

29. Network Schedule Risk Analysis
2, p = 0.5
6, p = 0.5
. . .
4 months
. . . 4
4 Equally Likely Outcomes 2 2
EV = 4 months 2 6
_20_ 4 6
EV =
= 5 months 6 2 6 6
_6_ 20
4/17/2017

30. Cost Risk Analysis: COCOMO II
Cone of Uncertainty methods and estimate ranges
Applications Composition, Early Design, Post-Architecture
Nature of COCOMO II Model
Size parameters, scale factors, cost drivers
Sensitivity and tradeoff analyses
Monte Carlo analysis
Enter size, scale factor, cost driver uncertainty ranges
Run model many times with random parameter selections within ranges
Produces cumulative probability distribution curve
4/17/2017

31. COCOMO Baseline Overview I
Software product size estimate
Software development mainten ance cost and schedule estimates
Software product, process, com-
puter, and personal attributes
Cost, schedule, distribution by
Software reuse, maintenance,
and increment parameters
phase, activity, increment
Software organization’s
Project data
COCOMO recalibrated to organization’s data
COCOMO
4/17/2017

32. COCOMO II Model Stages
4/17/2017

33. Factors for Project Scaling Effects
Nominal person-months = A*(size)B
B = (scale factor ratings)
- B ranges from 0.91 to 1.23
- 5 scale factors; 6 rating levels each
Scale factors:
- Precedentedness (PREC)
- Development flexibility (FLEX)
- Architecture/ risk resolution (RESL)
- Team cohesion (TEAM)
- Process maturity (PMAT, derived from SEI CMM)
4/17/2017

34. Rating the Scale Factors(
Size ) B EM i Õ æ è ç ö ø ÷ PM =
2.94
estimated B =
0.91 . + . 01 å SF i
Scale Factors ( Wi )
Very Low
Low
Nominal
High
Very High
Extra High
PREC
thoroughly
unprecedented
largely
somewhat
generally
familiar
largely familiar
throughly
FLEX
rigorous
occasional
relaxation
some
general
conformity
general goals
RESL
little (20%)
some (40%)
often (60%)
(75%)
mostly (90%)
full (100%)
TEAM
very difficult
interactions
some difficult
basically
cooperative
highly
seamless
PMAT
weighted sum of 18 KPA achievement levels
4/17/2017

35. COCOMO II. 2000 Productivity Ranges
Scale Factor Ranges: 10, 100, 1000 KSLOC
Development Flexibility (FLEX)
Team Cohesion (TEAM)
Develop for Reuse (RUSE)
Precedentedness (PREC)
Architecture and Risk Resolution (RESL)
Platform Experience (PEXP)
Data Base Size (DATA)
Required Development Schedule (SCED)
Language and Tools Experience (LTEX)
Process Maturity (PMAT)
Storage Constraint (STOR)
Use of Software Tools (TOOL)
Platform Volatility (PVOL)
Applications Experience (AEXP)
Multi-Site Development (SITE)
Documentation Match to Life Cycle Needs (DOCU)
Required Software Reliability (RELY)
Personnel Continuity (PCON)
Time Constraint (TIME)
Programmer Capability (PCAP)
Analyst Capability (ACAP)
Product Complexity (CPLX) 1
1.2
1.4
1.6
1.8 2
2.2
2.4
Productivity Range
4/17/2017

36. USC COCOMO II Model
4/17/2017

37. Tradeoffs Among Cost, Schedule, and Reliability: COCOMO II
(RELY, MTBF (hours))
For 100-KSLOC set of features
Can “pick all three” with 77-KSLOC set of features
-- Cost/Schedule/RELY:
“pick any two” points
4/17/2017

38. COCOMO II Estimation Accuracy:
Percentage of sample projects within 30% of actuals
-Without and with calibration to data source
COCOMO81
COCOMOII.2000
COCOMOII.1997
# Projects 63 83
161
Effort
Schedule
81%
52%
64%
75%
80%
61%
62%
72%
65%
4/17/2017

39. Software Risk Management
4/17/2017

40. Risk Exposure Factors and Contours: Satellite Experiment Software
4/17/2017

41. Risk Management Options
Buying information
C: Prototype fault tolerance features’ effect on performance
Risk avoidance
Risk transfer
Risk reduction
Risk acceptance
4/17/2017

42. Risk Management Plans
For Each Risk Item, Answer the Following Questions:
1. Why?
Risk Item Importance, Relation to Project Objectives
2. What, When?
Risk Resolution Deliverables, Milestones, Activity Nets
3. Who, Where?
Responsibilities, Organization
4. How?
Approach (Prototypes, Surveys, Models, …)
5. How Much?
Resources (Budget, Schedule, Key Personnel)
4/17/2017

43. Risk Management Plan: Fault Tolerance Prototyping
1. Objectives (The “Why”)
Determine, reduce level of risk of the software fault tolerance features causing unacceptable performance
Create a description of and a development plan for a set of low-risk fault tolerance features
2. Deliverables and Milestones (The “What” and “When”)
By week 3
1. Evaluation of fault tolerance option
2. Assessment of reusable components
3. Draft workload characterization
4. Evaluation plan for prototype exercise
5. Description of prototype
By week 7
6. Operational prototype with key fault tolerance features
7. Workload simulation
8. Instrumentation and data reduction capabilities
9. Draft Description, plan for fault tolerance features
By week 10
10. Evaluation and iteration of prototype
11. Revised description, plan for fault tolerance features
4/17/2017

44. Risk Management Plan: Fault Tolerance Prototyping (concluded)
Responsibilities (The “Who” and “Where”)
System Engineer: G. Smith
Tasks 1, 3, 4, 9, 11, support of tasks 5, 10
Lead Programmer: C. Lee
Tasks 5, 6, 7, 10 support of tasks 1, 3
Programmer: J. Wilson
Tasks 2, 8, support of tasks 5, 6, 7, 10
Approach (The “How”)
Design-to-Schedule prototyping effort
Driven by hypotheses about fault tolerance-performance effects
Use real-time OS, add prototype fault tolerance features
Evaluate performance with respect to representative workload
Refine Prototype based on results observed
Resources (The “How Much”)
$60K - Full-time system engineer, lead programmer, programmer (10 weeks)*(3 staff)*($2K/staff-week)
$0K - 3 Dedicated workstations (from project pool)
$0K - 2 Target processors (from project pool)
$0K - 1 Test co-processor (from project pool)
$10K - Contingencies
$70K - Total
4/17/2017

45. Risk Monitoring Milestone Tracking Top-10 Risk Item Tracking
Monitoring of Risk Management Plan Milestones
Top-10 Risk Item Tracking
Identify Top-10 risk items
Highlight these in monthly project reviews
Focus on new entries, slow-progress items
Focus review on manger-priority items
Risk Reassessment
Corrective Action
4/17/2017

46. Project Top 10 Risk Item List: Satellite Experiment Software
Mo. Ranking
This Last #Mo.
Risk Item
Risk Resolution Progress
Replacing Sensor-Control Software Top Replacement Candidate Unavailable
Developer
Target Hardware Delivery Delays Procurement Procedural Delays
Sensor Data Formats Undefined Action Items to Software, Sensor Teams;
Due Next Month
Staffing of Design V&V Team Key Reviewers Committed; Need Fault-
Tolerance Reviewer
Software Fault-Tolerance May Fault Tolerance Prototype Successful
Compromise Performance
Accommodate Changes in Data Meeting Scheduled With Data Bus
Bus Design Designers
Testbed Interface Definitions Some Delays in Action Items; Review
Meeting Scheduled
User Interface Uncertainties User Interface Prototype Successful
TBDs In Experiment Operational TBDs Resolved
Concept
Uncertainties In Reusable Required Design Changes Small,
Monitoring Software Successfully Made
4/17/2017

47. Conclusions Risk management starts on Day One
Delay and denial are serious career risks
Data provided to support early investment
ICM provides process framework for early risk resolution
Stakeholder identification and win condition reconciliation
Evidence-risk-driven commitment milestones
Risk analysis helps determine “how much is enough”
Testing, planning, specifying, prototyping,…
Balance risks of doing too little, too much
4/17/2017