1. University of Southern California Center for Systems and Software Engineering Reducing Estimation Uncertainty with Continuous Assessment Framework Pongtip Aroonvatanaporn Annual Research Review March 8, 2011

2. University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/20112© USC-CSSE

3. University of Southern California Center for Systems and Software Engineering The Cone of Uncertainty 03/08/20113© USC-CSSE Inexperienced teams Experienced teams Also applies to project estimation accuracy

4. University of Southern California Center for Systems and Software Engineering Definition Inexperience –Inexperienced in general –Experienced, but in a new domain –Anything that is new with little knowledge or experience 03/08/2011© USC-CSSE4

5. University of Southern California Center for Systems and Software Engineering The Problem Experienced teams can produce better estimates –Use “yesterday’s weather” –Past projects of comparable size –Past data of team’s productivity –Knowledge of accumulated problems and solutions Inexperienced teams do not have this luxury No tools or data that monitors project’s progression within the cone of uncertainty 03/08/20115© USC-CSSE

6. University of Southern California Center for Systems and Software Engineering The Problem Imprecise project scoping –Overestimate vs. underestimate Manual assessments are tedious –Complex and discouraging Project estimation not revisited –Insufficient data to perform predictions –Project’s uncertainties not adjusted Limitations in software cost estimation –Models cannot fully compensate for lack of knowledge and understanding 03/08/20116© USC-CSSE

7. University of Southern California Center for Systems and Software Engineering The Goal Develop a framework to address mentioned issues Help unprecedented projects track project progression Reduce the uncertainties in estimation –Achieve eventual convergence of estimate and actual Must be quick and easy to use 03/08/20117© USC-CSSE

8. University of Southern California Center for Systems and Software Engineering Benefits Improve project planning and management –Resources and goals Improved product quality control Actual project progress tracking –Better understanding of project status –Actual progress reports 03/08/20118© USC-CSSE

9. University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/20119© USC-CSSE

10. University of Southern California Center for Systems and Software Engineering Estimation Model Integration of the Unified Code Count tool and COCOMO II estimation model 03/08/201110© USC-CSSE Adjusted with REVL

11. University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/201111© USC-CSSE

12. University of Southern California Center for Systems and Software Engineering Experiment Setup Performed simulation on 2 projects from USC software engineering course Project similarities –Real-client –SAIV: 24-weeks –Architected agile process, 8-member team –Size, type, and complexities –Product E-services Web content management system JSP, MySQL, Tomcat 03/08/201112© USC-CSSE

13. University of Southern California Center for Systems and Software Engineering Obtaining Data Source code files retrieved from Subversion server Simulation of assessment done weekly Both teams were closely involved –Provide estimation of module completion –Rationale 03/08/201113© USC-CSSE

14. University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/201114© USC-CSSE

15. University of Southern California Center for Systems and Software Engineering Results Accumulated effort Initial estimate Adjusted estimate ~50 % ~18% 03/08/201115© USC-CSSE

16. University of Southern California Center for Systems and Software Engineering Results Project progress reaches 100% –Reflects reality Estimation errors reduced to 0% 03/08/201116© USC-CSSE

17. University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending the framework 03/08/201117© USC-CSSE

18. University of Southern California Center for Systems and Software Engineering Conclusion Both teams demonstrated the same phenomenon –Gaps in estimation errors decrease –Representation of “cone of uncertainty” Estimation framework reflects the reality of project’s progress Assessment process was quick and simple –Requires few inputs –Little analysis needed Assessment framework help inexperienced team improve project tracking and estimation 03/08/201118© USC-CSSE

19. University of Southern California Center for Systems and Software Engineering Future Work Tool development currently in progress Determine the frequencies of assessments required –The sweet spot Observe prediction accuracies –Experiment on projects of larger scale –Experiment on projects of different types Use concept of value-based –Apply weights to calculation of each software module based on priorities and criticalities –How to adjust COCOMO parameters 03/08/201119© USC-CSSE

20. University of Southern California Center for Systems and Software Engineering Outline Introduction and Motivation Framework Model Experiment Results Conclusion and future work Extending to the Framework 03/08/201120© USC-CSSE

21. University of Southern California Center for Systems and Software Engineering Problem and Motivation No effective way to improve COCOMO estimation accuracy –How to adjust the parameters? –Often done by guessing, not accurate, not realistic Need an approach to assess team performance –Good performance is essential in achieving successful project –Aid in stabilization and synchronization of the team 03/08/2011© USC-CSSE21

22. University of Southern California Center for Systems and Software Engineering Approach Develop an assessment framework to aid COCOMO estimation Determine the criteria for adjusting COCOMO parameters Develop a survey-based approach to implement the framework –Questions correlate to COCOMO cost drivers and scale factors 03/08/2011© USC-CSSE22

23. University of Southern California Center for Systems and Software Engineering Survey Question Development Team members assess and evaluate their own team –operational concept engineering, requirements gathering, business analysis, collaboration, etc. Identify strengths, weaknesses, issues. –Things done right and things done wrong Survey questions derived based on these assessment data 03/08/2011© USC-CSSE23 Team’s assessment performed by each team member Strengths, weaknesses, issues, etc. Survey questions

24. University of Southern California Center for Systems and Software Engineering Question  COCOMO How to determine the effects? –Which COCOMO parameters are affected by the question? –How much? Two approaches: –Finding correlation –Expert advice 03/08/2011© USC-CSSE24

25. University of Southern California Center for Systems and Software Engineering Team Performance Correlation 03/08/2011© USC-CSSE25 Issues filed by IV&Vers Team grades Client satisfaction Estimated effort from COCOMO Actual effort reported by team Survey questions Survey Compare Correlation

26. University of Southern California Center for Systems and Software Engineering Expert Advice Survey experts for opinions Provide ratings for weights on effects –Relationship between question and COCOMO parameters –Some questions affect COCOMO drivers more than others Ratings can be averaged to determine actual weights 03/08/2011© USC-CSSE26

27. University of Southern California Center for Systems and Software Engineering Expert Advice SubjectQuestionProductPlatformPersonnelProject RELYDATADOCUCPLXRUSETIMESTORPVOLACAPAPEXPCAPPLEXLTEXPCONTOOLSCEDSITE Prototyping Do you prototype regularly to verify the validity of requirements among all the stakeholders?0.5NA0.70.40.3NA 0.80.40.2 NA0.20.4NA Prototyping Do the prototypes address the critical use- cases, GUI issues, and risks and evolve over time as requirements change?0.9NA0.20.40.7NA 0.20.80.40.20.4 NA 0.5 Business process Do you understand the current and proposed business process or workflow? This includes the current system that is being utilized and what needs to be improved0.3NA 0.80.4NA 0.7 03/08/2011© USC-CSSE27 Relationship between question/answer and COCOMO parameter it affects

28. University of Southern California Center for Systems and Software Engineering Benefit Improved process to adjusting project estimates –Adjustments made based on data, not guesses Faster convergence of “Cone of Uncertainty” –Less range of possible product outcomes –More accurate project tracking 03/08/2011© USC-CSSE28

29. University of Southern California Center for Systems and Software Engineering Tool Support Utilize IBM Jazz Server technology –Focus on distributed environment support –Provides services for user management, team management, project management Integration of Unified Code Count tool and COCOMO model –Automatic code counting –Effort calculation –COCOMO parameter adjustments 03/08/2011© USC-CSSE29

30. University of Southern California Center for Systems and Software Engineering Conclusion A semi-automated framework to reduce estimation uncertainty throughout project life cycle Integration of Code Count and COCOMO –Helps track project progress –Shows improved estimates Survey-based assessment –Aid in team synchronization and stabilization –Automatically adjust COCOMO parameters to reflect reality of project 03/08/2011© USC-CSSE30

31. University of Southern California Center for Systems and Software Engineering Questions? 03/08/2011© USC-CSSE31

32. University of Southern California Center for Systems and Software Engineering Backup Slides 03/08/2011© USC-CSSE32

33. University of Southern California Center for Systems and Software Engineering 03/08/2011© USC-CSSE33

34. University of Southern California Center for Systems and Software Engineering 03/08/2011© USC-CSSE34

35. University of Southern California Center for Systems and Software Engineering 03/08/2011© USC-CSSE35

36. University of Southern California Center for Systems and Software Engineering 03/08/2011© USC-CSSE36

37. University of Southern California Center for Systems and Software Engineering 03/08/2011© USC-CSSE37

38. University of Southern California Center for Systems and Software Engineering References 1.Boehm, B., Abts, C., Brown, A. W., Chulani, S., Clark, B. K., Horowitz, E., Madachy, R., Reifer, D. J., and Steece, B. Software Cost Estimation with COCOMO II, Prentice-Hall, 2000. 2.Cohn, M. Agile Estimating and Planning, Prentice-Hall, 2005 3.DeMarco, T. Controlling Software Projects: Management, Measurement, and Estimation, Yourdon Press, 1982 4.Fleming, Q. W. and Koppelman, J. M. Earned Value Project Management, 2 nd edition, Project Management Institute, 2000 5.Jorgensen, M. and Boehm, B. “Software Development Effort Estimation: Formal Models or Expert Judgment?” IEEE Software, March-April 2009, pp. 14-19 6.Nguyen, V., Deeds-Rubin, S., Tan, T., and Boehm, B. "A SLOC Counting Standard," COCOMO II Forum 2007 7.Stutzke, R. D. Estimating Software-Intensive Systems, Pearson Education, Inc, 2005. 03/08/201138© USC-CSSE

39. University of Southern California Center for Systems and Software Engineering Related Work Software estimation methods –Estimating Software-Intensive Systems [Stutzke, 2005] –Expert-judgement vs. parametric-model [Jorgensen, 2007] –Agile estimation [Cohn, 2005] Software estimation uncertainty –PERT sizing methods [Nguyen, 2007] –Wideband Delphi estimate distributions [Boehm, 2000] Software project tracking methods –Controlling Software Projects [DeMarco, 1982] –Earned Value Management [Fleming, 2000] 03/08/201139© USC-CSSE