© 2011 Carnegie Mellon University QUELCE: Quantifying Uncertainty in Early Lifecycle Cost Estimation Presenters:Dave Zubrow PhD Bob Ferguson (SEMA) Date:November.

Slides:



Advertisements
Similar presentations
Project Management Concepts
Advertisements

Cost as a Business Driver 1 John Brown C Eng MIEE mr_ Software Cost Estimation.
Carnegie Mellon University Software Engineering Institute CERT® Knowledgebase Copyright © 1997 Carnegie Mellon University VU#14202 UNIX rlogin with stack.
© 2013 Carnegie Mellon University UFO: From Underapproximations to Overapproximations and Back! Arie Gurfinkel (SEI/CMU) with Aws Albarghouthi and Marsha.
UNCLASSIFIED © 2011 Carnegie Mellon University Building Malware Infection Trees Jose Andre Morales 1, Michael Main 2, Weilang Luo 3, Shouhuai Xu 2,3, Ravi.
© 2014 Microsoft Corporation. All rights reserved.
© 2011 Carnegie Mellon University System of Systems V&V John B. Goodenough October 19, 2011.
Chapter 2 The Software Process
© 2010 Carnegie Mellon University B OXES : A Symbolic Abstract Domain of Boxes Arie Gurfinkel and Sagar Chaki Software Engineering Institute Carnegie Mellon.
© 2013 Carnegie Mellon University Academy for Software Engineering Education and Training, 2013 Session Architect: Tony Cowling Session Chair: Nancy Mead.
2013 COCOMO Forum Stoddard, 24 October 2013 © 2013 Carnegie Mellon University Harvesting Reference Points for Cost Estimation: A Step in the SEI’s Cost.
© 2013 Carnegie Mellon University Measuring Assurance Case Confidence using Baconian Probabilities Charles B. Weinstock John B. Goodenough Ari Z. Klein.
© 2010 Carnegie Mellon University ® CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University. V&V Principles Verification.
© Carnegie Mellon University The CERT Insider Threat Center.
© 2012 Carnegie Mellon University UFO: Verification with Interpolants and Abstract Interpretation Arie Gurfinkel and Sagar Chaki Software Engineering Institute.
© 2010 Carnegie Mellon University Acquisition Implications of SOA Adoption Software Engineering Institute Carnegie Mellon University Pittsburgh, PA
© 2015 Carnegie Mellon University Property Directed Polyhedral Abstraction Nikolaj Bjørner and Arie Gurfinkel VMCAI 2015.
Copyright Robust Decisions Inc Decision Management in Product Development Dr. David Ullman President Robust Decisions, Inc.
CS351 © 2003 Ray S. Babcock Cost Estimation ● I've got Bad News and Bad News!
© 2011 Carnegie Mellon University Should-Cost: A Use for Parametric Estimates Additional uses for estimation tools Presenters:Bob Ferguson (SEMA) Date:November.
Project Risk Management Risk Mitigation. Risk Management  The prime objective of risk management is to minimize the impact and probability of the occurrence.
Pre-Project Planning Lessons from the Construction Industry Institute Construction Industry Institute Michael Davis, P. Eng, PMP Ontario Power Generation.
Capability Maturity Model
Release & Deployment ITIL Version 3
Ipek Ozkaya, COCOMO Forum © 2012 Carnegie Mellon University Affordability and the Value of Architecting Ipek Ozkaya Research, Technology.
© 2010 Carnegie Mellon University Team Software Process.
© VESP International Pty Limited To Contents Slide CLICK to advance slides/ bullet points within slides Integrated Master Planner An Overview.
N By: Md Rezaul Huda Reza n
VTT-STUK assessment method for safety evaluation of safety-critical computer based systems - application in BE-SECBS project.
Conditions and Terms of Use
© 2012 Microsoft Corporation. All rights reserved.
SIMULATION USING CRYSTAL BALL. WHAT CRYSTAL BALL DOES? Crystal ball extends the forecasting capabilities of spreadsheet model and provide the information.
Service Transition & Planning Service Validation & Testing
Basic of Project and Project Management Presentation.
Business Information Management II. “Copyright and Terms of Service Copyright © Texas Education Agency. The materials found on this website are copyrighted.
Ahmad Al-Ghoul. Learning Objectives Explain what a project is,, list various attributes of projects. Describe project management, discuss Who uses Project.
Your measurement technology partner for global competitiveness CSIR National Metrology Laboratory The National Metrology Laboratory of South Africa Feedback.
Welcome to Session 3 – Project Management Process Overview
1 Introduction to Software Engineering Lecture 1.
Software Engineering - I
Develop Project Charter
1 DISTRIBUTION A. Approved for Public Release; Distribution Unlimited. 88ABW , 23 May Integrity  Service  Excellence ADT 101: Introduction.
Project Portfolio Management Business Priorities Presentation.
SOFTWARE PROJECT MANAGEMENT
Business Management. 2 “Copyright and Terms of Service Copyright © Texas Education Agency. The materials found on this website are copyrighted © and trademarked.
Virtual Business Virtual Communication Copyright © Texas Education Agency, All rights reserved.
Author Software Engineering Institute
COCOMO Forum 2015 Bob Ferguson © 2015 Carnegie Mellon University Distribution Statement A: Approved for Public Release; Distribution is Unlimited Making.
CPSC 871 John D. McGregor Module 6 Session 2 Validation and Verification.
© 2015 Carnegie Mellon University COCOMO 2015 November 17, 2015 Distribution Statement A: Approved for Public Release; Distribution is Unlimited Causal.
Advanced Project Management Project Planning Phase Ghazala Amin.
SCOPE DEFINITION,VERIFICATION AND CONTROL Ashima Wadhwa.
1 CERT BFF: From Start To PoC June 09, 2016 © 2016 Carnegie Mellon University This material has been approved for public release and unlimited distribution.
Information Technology Project Management, Seventh Edition.
Deploying an Intelligent Pairing Assistant for Air Operation Centers Jeremy Ludwig, Ph.D. June 21, Distribution A: Approved for public release.
Secure Software Workforce Development Panel Session
Chapter 33 Estimation for Software Projects
Author Software Engineering Institute
Michael Spiegel, Esq Timothy Shimeall, Ph.D.
ISA 201 Intermediate Information Systems Acquisition
Software Engineering: A Practitioner’s Approach, 6/e Chapter 23 Estimation for Software Projects copyright © 1996, 2001, 2005 R.S. Pressman & Associates,
Metrics-Focused Analysis of Network Flow Data
What is Project Cost Management?
QUELCE: Quantifying Uncertainty in Early Lifecycle Cost Estimation
Dynamic Cyber Training with Moodle
Chapter 33 Estimation for Software Projects
Capability Maturity Model
Software Engineering: A Practitioner’s Approach, 6/e Chapter 23 Estimation for Software Projects copyright © 1996, 2001, 2005 R.S. Pressman & Associates,
Capability Maturity Model
Metrics That Work for You
Presentation transcript:

© 2011 Carnegie Mellon University QUELCE: Quantifying Uncertainty in Early Lifecycle Cost Estimation Presenters:Dave Zubrow PhD Bob Ferguson (SEMA) Date:November 3, 2011 Location:COCOMO Forum 2011

2 © 2011 Carnegie Mellon University DOD Acquisition Process and GAO Knowledge- Based Acquisition Practices Source: GAO T “Establishing realistic cost and schedule estimates that are matched to available resources: Cost and schedule estimates are often based on overly optimistic assumptions. Our previous work shows that without the ability to generate reliable cost estimates, programs are at risk of experiencing cost overruns, missed deadlines, and performance shortfalls. Inaccurate estimates do not provide the necessary foundation for sufficient funding commitments. Engineering knowledge is required to achieve more accurate, reliable cost estimates at the outset of a program.

3 © 2011 Carnegie Mellon University DOD Acquisition Process and GAO Knowledge- Based Acquisition Practices Source: GAO T

4 © 2011 Carnegie Mellon University ABC Approval Acquisition Phases and Decision Milestones Technology Development Engineering & Manufacturing Production & Deployment N Cost Estimate Based on: Limited Information Expert Judgment Analogies Est. $$$Est. $$Est. $ Delay Y Materiel Solution Challenges: 1) change and uncertainty 2) optimistic judgment Estimating the Cost of Development at Milestone A

5 © 2011 Carnegie Mellon University QUELCE for Producing Milestone A Estimates Brainstorm Change Drivers and Define States Develop Cause and Effect Matrix of Change Drivers Rate Relationships, Restructure and Reduce using DSM Produce BBN Model of Reduced Matrix Assign Probabilities and Conditional Probabilities to Nodes in BBN Define Scenarios of Program Execution Use Monte Carlo to Select Combinations of BBN Outputs to Produce Cost Estimate Distributions Map BBN Change Factor Output States to COCOMO Cost Driver Values

6 © 2011 Carnegie Mellon University Change Drivers and States Factors seeded by, but not limited to, Probability of Program Success (PoPS) factors.

7 © 2011 Carnegie Mellon University Cause and Effect Matrix for Change Drivers Each cell gets a value (blank, 1, 2, or 3) to reflect the perceived cause-effect relationship of the row heading to the column heading) Note: The sum of a column represents a dependency score for the column header. The sum of a row is the value of the driving force of the row header

8 © 2011 Carnegie Mellon University Reduced Cause and Effect Matrix Use Design Structure Matrix techniques to reduce so an acyclic graph can be produced. Indicates remaining cycle that must be removed

9 © 2011 Carnegie Mellon University BBN of Reduced Cause and Effect Matrix Translate the C-E Matrix into a BBN. Orange nodes are program change factors. Green nodes are outputs that will link to COCOMO cost drivers. These output nodes were selected as an example and represent sets of COCOMO cost drivers.

10 © 2011 Carnegie Mellon University Assign Probabilities and Conditional Probabilities to BBN Nodes Use expert judgment to assign probabilities and conditional probabilities to the nodes. These assignments could also be empirically based if the data are available. Capability Definition is affected by CONOPS and Strategic Vision

11 © 2011 Carnegie Mellon University Define Scenarios of Program Execution Scenarios for alternate futures specify nominal or non-nominal states for selected change drivers to test alternative results.

12 © 2011 Carnegie Mellon University Map BBN Change Factor Output States to COCOMO Cost Driver Values BBN output states are mapped to values of COCOMO cost drivers. Currently done with expert judgment. Later could be done using a data-based algorithm. Distributions of BBN outputs used in next step.

13 © 2011 Carnegie Mellon University Use Monte Carlo to Select Combinations of BBN Outputs to Produce Cost Estimate Distributions Using distribution of BBN outputs, Monte Carlo simulation is used to produce the distribution of the cost estimation for each defined scenario. Mapped COCOMO value 4 4 BBN Outputs

14 © 2011 Carnegie Mellon University Task 2: Develop Efficient Techniques To Calibrate Expert Judgment of Program Uncertainties Solution Calibrated Un-Calibrated Estimate of SW Size Domain-Specific “reference points” 1)Size of ground combat vehicle targeting feature xyz in 2002 consisted of 25 KSLOC Ada 2)Size of Army artillery firing capability feature abc in 2007 consisted of 18 KSLOC C++ 3)… Step 1: Expert virtual training of “reference points” Step 2: Expert takes series of domain specific tests Step 3: Expert reduces overconfidence Step 4: Expert renders calibrated estimate of size Calibrated = more realistic size and wider range to reflect true expert uncertainty

15 © 2011 Carnegie Mellon University Next Steps Classroom experiments with software & systems engineering graduate students Structured feedback for early validation & refinement of our method, initially: – Selected steps at least through reducing the client’s dependency matrix at the University of Arizona – Calibrating expert judgment & reconciliation of differences in judgment at Carnegie Mellon University Invitation to Participate Many pieces and parts to test and improve Empirically validating the overall approach – Retrospective: projects that were estimated and have recorded change history Testing BBN output parameters and mapping to estimation parameters Fitting to additional estimation tools and cost estimation relationships (CER) Do new estimates properly inform decisions about risk and change management? (secondary benefit)

16 © 2011 Carnegie Mellon University Contact Information Presenters/ Points of Contact Dave Zubrow SEMA Telephone: Bob Ferguson SEMA Telephone: Web: U.S. mail: Software Engineering Institute Customer Relations 4500 Fifth Avenue Pittsburgh, PA USA Customer Relations Telephone: SEI Phone: SEI Fax:

17 © 2011 Carnegie Mellon University NO WARRANTY THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. Use of any trademarks in this presentation is not intended in any way to infringe on the rights of the trademark holder. This Presentation may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at This work was created in the performance of Federal Government Contract Number FA C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. The Government of the United States has a royalty-free government-purpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have or permit others to do so, for government purposes pursuant to the copyright license under the clause at