University of Southern California Center for Systems and Software Engineering Reducing Estimation Uncertainty with Continuous Assessment: Tracking the “Cone of Uncertainty” Pongtip Aroonvatanaporn, Chatchai Sinthop, Barry Boehm {aroonvat, sinthop, November 2, 2010

University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work 11/02/20102© USC-CSSE

University of Southern California Center for Systems and Software Engineering The Cone of Uncertainty 11/02/20103© USC-CSSE Inexperienced teams Experienced teams Also applies to project estimation accuracy

University of Southern California Center for Systems and Software Engineering Definition Inexperience –Inexperienced in general –Experienced, but in a new domain –Anything that is new with little knowledge or experience 11/02/2010© USC-CSSE4

University of Southern California Center for Systems and Software Engineering The Problem Experienced teams can produce better estimates –Use “yesterday’s weather” –Past projects of comparable size –Past data of team’s productivity –Knowledge of accumulated problems and solutions Inexperienced teams do not have this luxury No tools or data that monitors project’s progression within the cone of uncertainty 11/02/20105© USC-CSSE

University of Southern California Center for Systems and Software Engineering The Problem Imprecise project scoping –Overestimate vs. underestimate Manual assessments are tedious –Complex and discouraging Project estimation not revisited –Insufficient data to perform predictions –Project’s uncertainties not adjusted Limitations in software cost estimation –Models cannot fully compensate for lack of knowledge and understanding 11/02/20106© USC-CSSE

University of Southern California Center for Systems and Software Engineering The Goal Develop a framework to address mentioned issues Help unprecedented projects track project progression Reduce the uncertainties in estimation –Achieve eventual convergence of estimate and actual Must be quick and easy to use 11/02/20107© USC-CSSE

University of Southern California Center for Systems and Software Engineering Benefits Improve project planning and management –Resources and goals Improved product quality control Actual project progress tracking –Better understanding of project status –Actual progress reports 11/02/20108© USC-CSSE

University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work 11/02/20109© USC-CSSE

University of Southern California Center for Systems and Software Engineering Estimation Model Integration of the Unified Code Count tool and COCOMO II estimation model 11/02/201010© USC-CSSE Adjusted with REVL

University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work 11/02/201011© USC-CSSE

University of Southern California Center for Systems and Software Engineering Experiment Setup Performed simulation on 2 projects from USC software engineering course Project similarities –Real-client –SAIV: 24-weeks –Architected agile process, 8-member team –Size, type, and complexities –Product E-services Web content management system JSP, MySQL, Tomcat 11/02/201012© USC-CSSE

University of Southern California Center for Systems and Software Engineering Obtaining Data Source code files retrieved from Subversion server Simulation of assessment done weekly Both teams were closely involved –Provide estimation of module completion –Rationale 11/02/201013© USC-CSSE

University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work 11/02/201014© USC-CSSE

University of Southern California Center for Systems and Software Engineering Results Accumulated effort Initial estimate Adjusted estimate ~50 % ~18% 11/02/201015© USC-CSSE

University of Southern California Center for Systems and Software Engineering Results Project progress reaches 100% –Reflects reality Estimation errors reduced to 0% 11/02/201016© USC-CSSE

University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work 11/02/201017© USC-CSSE

University of Southern California Center for Systems and Software Engineering Conclusion Both teams demonstrated the same phenomenon –Gaps in estimation errors decrease –Representation of “cone of uncertainty” Estimation framework reflects the reality of project’s progress Assessment process was quick and simple –Requires few inputs –Little analysis needed Assessment framework help inexperienced team improve project tracking and estimation 11/02/201018© USC-CSSE

University of Southern California Center for Systems and Software Engineering Future Work Tool development currently in progress Determine the frequencies of assessments required –The sweet spot Observe prediction accuracies –Experiment on projects of larger scale –Experiment on projects of different types Use concept of value-based –Apply weights to calculation of each software module based on priorities and criticalities –How to adjust COCOMO parameters 11/02/201019© USC-CSSE

University of Southern California Center for Systems and Software Engineering References 1.Boehm, B., Abts, C., Brown, A. W., Chulani, S., Clark, B. K., Horowitz, E., Madachy, R., Reifer, D. J., and Steece, B. Software Cost Estimation with COCOMO II, Prentice-Hall, Cohn, M. Agile Estimating and Planning, Prentice-Hall, DeMarco, T. Controlling Software Projects: Management, Measurement, and Estimation, Yourdon Press, Fleming, Q. W. and Koppelman, J. M. Earned Value Project Management, 2 nd edition, Project Management Institute, Jorgensen, M. and Boehm, B. “Software Development Effort Estimation: Formal Models or Expert Judgment?” IEEE Software, March-April 2009, pp Nguyen, V., Deeds-Rubin, S., Tan, T., and Boehm, B. "A SLOC Counting Standard," COCOMO II Forum Stutzke, R. D. Estimating Software-Intensive Systems, Pearson Education, Inc, /02/201020© USC-CSSE

University of Southern California Center for Systems and Software Engineering Backup Slides 11/02/2010© USC-CSSE21

University of Southern California Center for Systems and Software Engineering Related Work Software estimation methods –Estimating Software-Intensive Systems [Stutzke, 2005] –Expert-judgement vs. parametric-model [Jorgensen, 2007] –Agile estimation [Cohn, 2005] Software estimation uncertainty –PERT sizing methods [Nguyen, 2007] –Wideband Delphi estimate distributions [Boehm, 2000] Software project tracking methods –Controlling Software Projects [DeMarco, 1982] –Earned Value Management [Fleming, 2000] 11/02/201022© USC-CSSE