Download presentation
Presentation is loading. Please wait.
Published byMaria das Graças Maranhão Aveiro Modified over 6 years ago
1
QUELCE: Quantifying Uncertainty in Early Lifecycle Cost Estimation
Author Software Engineering Institute 12/29/2018 QUELCE: Quantifying Uncertainty in Early Lifecycle Cost Estimation Presenters: Dave Zubrow PhD Bob Ferguson (SEMA) Date: November 3, 2011 Location: COCOMO Forum 2011
2
Problem: Estimating the Cost of Development at Milestone A
Sparse Cost Information Prior to the Technology Development Phase (TDP), DOD requires a program estimate for both the TDP and the full program lifecycle. Since both the technology and the development strategy are still unknown, we do not have access to the usual cost drivers with the precision desired for estimation. Lots of Information is Available but Not Utilized for Estimate A preliminary analysis of alternatives has been performed. Requirements are in the form of capabilities. Scoping information is known and there are a number of analogies available. The Information is Confounded The extra information is not in a form that is currently suited to solving the problem. The various factors are related to one another so it is difficult to see the interrelationships and construct a sensible estimate.
3
Estimating the Cost of Development at Milestone A
Author Program 12/29/2018 Acquisition Phases and Decision Milestones Challenge 1: change and uncertainty A Technology Development B Engineering & Manufacturing C Production & Deployment Develop Estimate Challenge 2: optimistic judgment Technology Development Strategy Materiel Solution Estimate Cost Approval Scope of Estimates Cost Overruns Joint Strike Fighter – 300% Future Combat System – 50% DDG1000 – Fewer ships GAO Systemic Impact - $295 B Denied Sparse Cost Information Prior to the Technology Development Phase (TDP), DOD requires a program estimate for both the TDP and the full program lifecycle. Since both the technology and the development strategy are still unknown, we do not have access to the usual cost drivers with the precision desired for estimation. Lots of Information is Available A preliminary analysis of alternatives has been performed. Requirements are in the form of capabilities. Scoping information is known and there are a number of analogies available. The Information is Confounded The extra information is not in a form that is currently suited to solving the problem. The various factors are related to one another so it is difficult to see the interrelationships and construct a sensible estimate. Why are we doing this? Challenges Model uncertainty associated with inputs Need to estimate earlier Sparse and uncertain information Uncalibrated expert judgment Delays Due to Reconciling Cost Estimates Ship to Shore Connector – 12 months Ground Combat Vehicle – 4 months + New AoA GAO Systemic Impact – 6 months average
4
Our Approach Develop Cause and Effect Matrix of Change Drivers
Author Program 12/29/2018 Brainstorm Change Drivers and Define States Develop Cause and Effect Matrix of Change Drivers Rate Relationships, Restructure and Reduce using DSM Produce BBN Model of Reduced Matrix Assign Probabilities and Conditional Probabilities to Nodes in BBN Define Scenarios of Program Execution Need to clarify terms: Change Drivers – categories of external and internal changes that affect program execution Cost Drivers – COCOMO cost drivers Use Monte Carlo to Select Combinations of BBN Outputs to Produce Cost Estimate Distributions Map BBN Change Factor Output States to COCOMO Cost Driver Values
5
Change Drivers and States
Factors seeded by, but not limited to, Probability of Program Success (PoPS) factors.
6
Cause and Effect Matrix for Change Drivers
Each cell gets a value (0, 1, 2, or 3) to reflect the perceived cause-effect relationship of the row heading to the column heading) Note: The sum of a column represents a dependency score for the column header. The sum of a row is the value of the driving force of the row header
7
Reduced Cause and Effect Matrix
Indicates remaining cycle that must be removed Use Design Structure Matrix techniques to reduce so an acyclic graph can be produced.
8
BBN of Reduced Cause and Effect Matrix
Translate the C-E Matrix into a BBN. Orange nodes are program change factors. Green nodes are outputs that will link to COCOMO cost drivers. These output nodes were selected as an example and represent sets of COCOMO cost drivers.
9
Assign Probabilities and Conditional Probabilities to BBN Nodes
Use expert judgment to assign probabilities and conditional probabilities to the nodes. These assignments could also be empirically based if the data are available. Capability Definition is affected by CONOPS and Strategic Vision
10
Define Scenarios of Program Execution
Scenarios for alternate futures specify nominal or non-nominal states for selected change drivers to test alternative results.
11
Map BBN Change Factor Output States to COCOMO Cost Driver Values
BBN output states are mapped to values of COCOMO cost drivers. Currently done with expert judgment. Later could be done using a data-based algorithm. Distributions of BBN outputs used in next step.
12
Use Monte Carlo to Select Combinations of BBN Outputs to Produce Cost Estimate Distributions
4 4 Mapped COCOMO value Using distribution of BBN outputs, Monte Carlo simulation is used to produce the distribution of the cost estimation for each defined scenario.
13
Completed Model includes BBN and Cost Estimation Tool
Author Program 12/29/2018 Tech. Dev. Strategy System Design Capability Analysis Mission & CONOPS KPPs Production Quantity Technical Challenge Acquisition Strategy Logistics & Support Project Challenge Contract Existing Cost Estimating Tools Size Growth Define change drivers and states Estimate conditional probability of change Develop scenarios Distribution for Cost Connect to estimating tool Simulate to establish ranges
14
Calibrating Expert Judgment of Program Uncertainties
Use calibration training with domain specific reference points to improve expert judgment for model inputs Problem 2: Calibrated experts reach agreement quicker! Domain-Specific cost “reference points” Un-Calibrated Calibrated Active & Completed MDAP Cost Experiences Independent Cost Review Results Probability of change Problem 1: Reference points prevent overly optimistic short duration estimates and less over-confidence with wider ranges to reflect their real knowledge!
15
Next Steps Classroom experiments with software & systems engineering graduate students Structured feedback for early validation & refinement of our method, initially: Selected steps at least through reducing the client’s dependency matrix at the University of Arizona Calibrating expert judgment & reconciliation of differences in judgment at Carnegie Mellon University Invitation to Participate Many pieces and parts to test and improve Empirically validating the overall approach Retrospective: projects that were estimated and have recorded change history Testing BBN output parameters and mapping to estimation parameters Fitting to additional estimation tools and cost estimation relationships (CER) Do new estimates properly inform decisions about risk and change management? (secondary benefit)
16
Author Software Engineering Institute
12/29/2018 Contact Information Presenters/ Points of Contact Dave Zubrow SEMA Telephone: Bob Ferguson Telephone: Web: U.S. mail: Software Engineering Institute Customer Relations 4500 Fifth Avenue Pittsburgh, PA USA Telephone: SEI Phone: SEI Fax:
17
Author Software Engineering Institute
NO WARRANTY THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. Use of any trademarks in this presentation is not intended in any way to infringe on the rights of the trademark holder. This Presentation may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at This work was created in the performance of Federal Government Contract Number FA C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. The Government of the United States has a royalty-free government-purpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have or permit others to do so, for government purposes pursuant to the copyright license under the clause at Author Software Engineering Institute 12/29/2018
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.