1. QUELCE: Quantifying Uncertainty in Early Lifecycle Cost Estimation
Author
Software Engineering Institute
12/29/2018
QUELCE: Quantifying Uncertainty in Early Lifecycle Cost Estimation
Presenters: Dave Zubrow PhD
Bob Ferguson (SEMA)
Date: November 3, 2011
Location: COCOMO Forum 2011

2. Problem: Estimating the Cost of Development at Milestone A
Sparse Cost Information
Prior to the Technology Development Phase (TDP), DOD requires a program estimate for both the TDP and the full program lifecycle. Since both the technology and the development strategy are still unknown, we do not have access to the usual cost drivers with the precision desired for estimation.
Lots of Information is Available but Not Utilized for Estimate
A preliminary analysis of alternatives has been performed. Requirements are in the form of capabilities. Scoping information is known and there are a number of analogies available.
The Information is Confounded
The extra information is not in a form that is currently suited to solving the problem. The various factors are related to one another so it is difficult to see the interrelationships and construct a sensible estimate.

3. Estimating the Cost of Development at Milestone A
Author
Program
12/29/2018
Acquisition Phases and Decision Milestones
Challenge 1: change and uncertainty A
Technology Development B
Engineering & Manufacturing C
Production & Deployment
Develop Estimate
Challenge 2: optimistic judgment
Technology Development Strategy
Materiel Solution
Estimate
Cost
Approval
Scope of Estimates
Cost Overruns
Joint Strike Fighter – 300%
Future Combat System – 50%
DDG1000 – Fewer ships
GAO Systemic Impact - $295 B
Denied
Sparse Cost Information
Prior to the Technology Development Phase (TDP), DOD requires a program estimate for both the TDP and the full program lifecycle. Since both the technology and the development strategy are still unknown, we do not have access to the usual cost drivers with the precision desired for estimation.
Lots of Information is Available
A preliminary analysis of alternatives has been performed. Requirements are in the form of capabilities. Scoping information is known and there are a number of analogies available.
The Information is Confounded
The extra information is not in a form that is currently suited to solving the problem. The various factors are related to one another so it is difficult to see the interrelationships and construct a sensible estimate.
Why are we doing this?
Challenges
Model uncertainty associated with inputs
Need to estimate earlier
Sparse and uncertain information
Uncalibrated expert judgment
Delays Due to Reconciling Cost Estimates
Ship to Shore Connector – 12 months
Ground Combat Vehicle – 4 months + New AoA
GAO Systemic Impact – 6 months average

4. Our Approach Develop Cause and Effect Matrix of Change Drivers
Author
Program
12/29/2018
Brainstorm Change Drivers and Define States
Develop Cause and Effect Matrix of Change Drivers
Rate Relationships, Restructure and Reduce using DSM
Produce BBN Model of Reduced Matrix
Assign Probabilities and Conditional Probabilities to Nodes in BBN
Define Scenarios of Program Execution
Need to clarify terms:
Change Drivers – categories of external and internal changes that affect program execution
Cost Drivers – COCOMO cost drivers
Use Monte Carlo to Select Combinations of BBN Outputs to Produce Cost Estimate Distributions
Map BBN Change Factor Output States to COCOMO Cost Driver Values

5. Change Drivers and States
Factors seeded by, but not limited to, Probability of Program Success (PoPS) factors.

6. Cause and Effect Matrix for Change Drivers
Each cell gets a value (0, 1, 2, or 3) to reflect the perceived cause-effect relationship of the row heading to the column heading)
Note: The sum of a column represents a dependency score for the column header. The sum of a row is the value of the driving force of the row header

7. Reduced Cause and Effect Matrix
Indicates remaining cycle that must be removed
Use Design Structure Matrix techniques to reduce so an acyclic graph can be produced.

8. BBN of Reduced Cause and Effect Matrix
Translate the C-E Matrix into a BBN. Orange nodes are program change factors. Green nodes are outputs that will link to COCOMO cost drivers.
These output nodes were selected as an example and represent sets of COCOMO cost drivers.

9. Assign Probabilities and Conditional Probabilities to BBN Nodes
Use expert judgment to assign probabilities and conditional probabilities to the nodes.
These assignments could also be empirically based if the data are available.
Capability Definition is affected by CONOPS and Strategic Vision

10. Define Scenarios of Program Execution
Scenarios for alternate futures specify nominal or non-nominal states for selected change drivers to test alternative results.

11. Map BBN Change Factor Output States to COCOMO Cost Driver Values
BBN output states are mapped to values of COCOMO cost drivers.
Currently done with expert judgment. Later could be done using a data-based algorithm.
Distributions of BBN outputs used in next step.

12. Use Monte Carlo to Select Combinations of BBN Outputs to Produce Cost Estimate Distributions4 4
Mapped COCOMO value
Using distribution of BBN outputs, Monte Carlo simulation is used to produce the distribution of the cost estimation for each defined scenario.

13. Completed Model includes BBN and Cost Estimation Tool
Author
Program
12/29/2018
Tech. Dev. Strategy
System Design
Capability Analysis
Mission & CONOPS
KPPs
Production Quantity
Technical Challenge
Acquisition Strategy
Logistics & Support
Project Challenge
Contract
Existing Cost Estimating Tools
Size Growth
Define change drivers and states
Estimate conditional probability of change
Develop scenarios
Distribution for Cost
Connect to estimating tool
Simulate to establish ranges

14. Calibrating Expert Judgment of Program Uncertainties
Use calibration training with domain specific reference points to improve expert judgment for model inputs
Problem 2: Calibrated experts reach agreement quicker!
Domain-Specific cost “reference points”
Un-Calibrated
Calibrated
Active & Completed MDAP Cost Experiences
Independent Cost Review Results
Probability of change
Problem 1: Reference points prevent overly optimistic short duration estimates and less over-confidence with wider ranges to reflect their real knowledge!

15. Next Steps
Classroom experiments with software & systems engineering graduate students
Structured feedback for early validation & refinement of our method, initially:
Selected steps at least through reducing the client’s dependency matrix at the University of Arizona
Calibrating expert judgment & reconciliation of differences in judgment at Carnegie Mellon University
Invitation to Participate
Many pieces and parts to test and improve
Empirically validating the overall approach
Retrospective: projects that were estimated and have recorded change history
Testing BBN output parameters and mapping to estimation parameters
Fitting to additional estimation tools and cost estimation relationships (CER)
Do new estimates properly inform decisions about risk and change management? (secondary benefit)

16. Author Software Engineering Institute
12/29/2018
Contact Information
Presenters/ Points of Contact
Dave Zubrow
SEMA
Telephone:
Bob Ferguson
Telephone:
Web:
U.S. mail:
Software Engineering Institute
Customer Relations
4500 Fifth Avenue
Pittsburgh, PA
USA
Telephone:
SEI Phone:
SEI Fax:

17. Author Software Engineering Institute
NO WARRANTY THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. Use of any trademarks in this presentation is not intended in any way to infringe on the rights of the trademark holder. This Presentation may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at This work was created in the performance of Federal Government Contract Number FA C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. The Government of the United States has a royalty-free government-purpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have or permit others to do so, for government purposes pursuant to the copyright license under the clause at
Author
Software Engineering Institute
12/29/2018