1. Technische Universität München The influence of software quality requirements on the suitability of software cost estimation methods 24th International Forum on COCOMO and Systems/Software Cost Modeling Marc Giombetti (TUM & MIT) Dr. Ricardo Valerdi (MIT) Dr. Stefan Wagner (TUM)

2. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 2 Outline Motivation Advantages and limitations of different software cost estimation methods Understanding & estimating software quality A controlled experiment – COCOMO II vs. Expert judgment Conclusions & Future work

3. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 3 Formal estimation models or expert judgment ? (1/2) M. Jørgensen and B. Boehm, “Software Development Effort Estimation: Formal Models or Expert Judgment?,” IEEE Software, vol. 26, 2009, pp. 14-19. Formal estimation models or expert judgment […] figure out when, and under what conditions, each method would be best.  This research adjusts this question and asks: When are formal models more useful than expert judgment; given a set of specific quality requirements.

4. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 4 Formal estimation models or expert judgment ? (2/2) Formal Estimation Models COCOMO, FP based models Mechanical quantification step Dependent on calibration data Objective & repeatable ? Industry usage? – Trust in a method which is not fully understood by PM? Expert judgment Checklist-based estimation, WBS, “gut feelings”, Judgment-based quantification step Dependent on individual estimator Biased towards optimism Only 15% of the journal papers analyzed judgment-based effort estimation * The difference between formal estimation models and expert judgment is not always clear  Combination of methods * M. Jørgensen and M. Shepperd, “A systematic review of software development cost estimation studies,” IEEE Transactions on Software Engineering, vol. 33, 2007, pp. 33-53.

5. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 5 Software Quality Software quality is an important factor for the success of software of any kind. –How much software quality assurance is enough? * –For several industries quality is not negociatable! One of the challenges of Software Quality is that "everyone feels they understand it“** Quality requirements need to be taken into consideration while estimateing the cost of future projects. * L. Huang and B. Boehm, “How Much Software Quality Investment Is Enough: A Value-Based Approach,” Software, IEEE, vol. 23, 2006, pp. 88-95. ** P.B. Crosby, Quality Is Free, Signet, 1980.

6. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 6 ISO/IEC 9126 - Software Quality Model International Organization for Standardization, ed., “ISO/IEC 9126-1:2001 - Software engineering - Product quality - Part 1: Quality model,” 2001.

7. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 7

8. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 8 The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (1/4) Estimation for development time (no schedule compression) Expert judgment : Top-down or bottom-up approach H1 H2

9. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 9 The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (2/4) System requirements and project settings –Domain: Financial application –Type: Web application –Development team size: 1 Project Manager 1 Senior developer, 1 QA, 3 Junior developers Expert estimation: Graduate students –Security: 5 students –Usability: 6 students

10. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 10 The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (3/4) Research hypotheses –H1 0 : In the case of additional security requirements There is no significant difference between the average development time estimated using expert estimation and the intermediate COCOMO II model estimate. –H2 0 : In the case of additional usability requirements: There is no significant difference between the average development time estimated using expert estimation and the intermediate COCOMO II model estimate. H1 H2

11. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 11 The influence of software quality requirements on expert estimation and COCOMO II – A controlled experiment (4/4) Experiment Goals –G.1. Analyze the relative impact of security and usability requirements on the development time of a web-application using intermediate COCOMO II. –G.2. Finding evidence if there is a significant difference between the schedule estimate for the web application enhanced by security requirements made using expert estimation compared to intermediate COCOMO II. –G.3. Finding evidence if there is a significant difference between the schedule estimate for the web application enhanced by usability requirements made using expert estimation compared to intermediate COCOMO II.

12. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 12 Experiment: COCOMO II Estimate

13. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 13 Experiment: Results (1/2) Usability N6 Outliers0 Median103 Mean108 St. Deviation17 Security N4 Outliers1 Median85,5 Mean85,25 St. Deviation24,15 H1 H2 Different mental models for the concepts security and usability???

14. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 14 Experiment Results (2/2) T-Test & Wilcoxon signed rank test at 5% significance level: –Reject H1  There IS a significant difference between the estimates of expert estimatiors and COCOMO II for the security enhancement. –Reject H2  There IS a significant difference between the estimates of expert estimatiors and COCOMO II for the usability enhancement. Threats to validity –Students in the role of expert estimators * –Classroom exercise: The requirements provided to the students were simplified, but chosen to be comparable to real requirements (in size & complexity). –Selection of quality requirements –Small dataset H1 H2 * P. Berander, “Using students as subjects in requirements prioritization,” Proceedings of the 2004 International Symposium on Empirical Software Engineering, IEEE Computer Society, 2004, pp. 167-176.

15. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 15 Conclusions & Future work Perception of quality attributes influences the estimators, but not the COCOMO model estimate. Human estimators underestimated security; overestimated usability Influence of quality for cost estimation - Improvements Conduction of experiment with more participants –Planned: Graduate students at TU Munich –Preferable: Software professionals with real estimation experience Focus on other quality aspects (maintainability, reliability,…) Conduction of a second experiment on the influence of usability (Focus on FP counting & COCOMO II)

16. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 16 Future work: COCOMO II – Estimating the influence of usability

17. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 17 Questions?

18. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 18 References M. Jørgensen and B. Boehm, “Software Development Effort Estimation: Formal Models or Expert Judgment?,” IEEE Software, vol. 26, 2009, pp. 14-19. M. Jørgensen and M. Shepperd, “A systematic review of software development cost estimation studies,” IEEE Transactions on Software Engineering, vol. 33, 2007, pp. 33-53. L. Huang and B. Boehm, “How Much Software Quality Investment Is Enough: A Value-Based Approach,” Software, IEEE, vol. 23, 2006, pp. 88-95. P.B. Crosby, Quality Is Free, Signet, 1980. International Organization for Standardization, ed., “ISO/IEC 9126-1:2001 - Software engineering - Product quality - Part 1: Quality model,” 2001. P. Berander, “Using students as subjects in requirements prioritization,” Proceedings of the 2004 International Symposium on Empirical Software Engineering, IEEE Computer Society, 2004, pp. 167-176.

19. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 19 Backup Slides

20. Technische Universität München Marc Giombetti The influence of software quality requirements on the suitability of software cost estimation methods 20 Expert estimation approach chosen by participants