1. smi COCOMO II Calibration Status COCOMO Forum October 2004

2. smi COCOMO Forum - October 20042 A Little History Calibration effort started in January 2002 Confusion –Repository in an inconsistent state –“Uncharacterized” data from many sources –Process for duplicating the 2000 calibration results –Schedule compression rating was inconsistent Expectation –New data had a lot of variation but… –Affiliates (and the user population in general) want an “Accurate” and up-to-date model – not just one that explained variation PRED(.25) versus R 2

3. smi COCOMO Forum - October 20043 Change in Approach Removed pre-1990 data from dataset used in calibration –This removed a lot of “converted” data Removed “bad” data –Incomplete: No duration data, estimated effort, no valid SLOC size Still use the Bayesian calibration approach developed by Chulani Changed to a holistic analysis approach: considered effort and duration together –Identified data that needed review –Schedule compression was automatically set

4. smi COCOMO Forum - October 20044 Post-1989 Data Using Current COCOMO II Values Effort Underestimated Duration Overestimated Effort Underestimated Duration Underestimated Effort Overestimated Duration Overestimated Effort Overestimated Duration Underestimated

5. smi COCOMO Forum - October 20045 Effort- Duration Error Interpretation Effort EstimatesDuration Estimates Data Validation / Interpretation Under-estimated Actual size data is too small due to reuse modeling Actual error and duration included lifecycle phases not in the model Difficult, low productivity projects Under-EstimatedOver-EstimatedSchedule Compression required Over-estimatedUnder-estimatedFixed-staffing levels Project slow-down Schedule Stretch-out Over-estimated Actual data is too large due to physical SLOC count, reuse modeling Actual effort and duration cover fewer lifecycle phases than estimated Easy, high productivity

6. smi COCOMO Forum - October 20046 Effort Estimate Error Compared to Size (Post 1989 – 89 Projects, 2000 Cal)

7. smi COCOMO Forum - October 20047 Duration Estimate Error Compared to Size (Post 1989 – 89 Projects, 2000 Cal)

8. smi COCOMO Forum - October 20048 Accuracy Results Effort Estimation Accuracy Duration Estimation Accuracy PRED 161 Dataset with 2000 Cal Values 89 Dataset with 2000 Cal Values 89 Dataset with 89 Cal Values 20637984 25688289 30758692 PRED 161 Dataset with 2000 Values 89 Dataset with 2000 Values 89 Dataset with 89 Values 20505664 25556371 30647082

9. smi COCOMO Forum - October 20049 Calibration Progress Reviewing new data –Dataset A: 8 projects –Dataset B: 52 projects –Dataset C: 13 projects –Dataset D: 4 projects –Dataset E: 10 projects –Dataset F: 8 projects

10. smi COCOMO Forum - October 200410 Dataset A Effort Underestimated Duration Overestimated Effort Underestimated Duration Underestimated Effort Overestimated Duration Overestimated Effort Overestimated Duration Underestimated

11. smi COCOMO Forum - October 200411 Dataset B Effort Underestimated Duration Overestimated Effort Underestimated Duration Underestimated Effort Overestimated Duration Overestimated Effort Overestimated Duration Underestimated

12. smi COCOMO Forum - October 200412 Dataset C Effort Underestimated Duration Overestimated Effort Underestimated Duration Underestimated Effort Overestimated Duration Overestimated Effort Overestimated Duration Underestimated

13. smi COCOMO Forum - October 200413 Dataset D & E Effort Underestimated Duration Overestimated Effort Overestimated Duration Overestimated Effort Overestimated Duration Underestimated

14. smi COCOMO Forum - October 200414 Dataset F Effort Underestimated Duration Overestimated Effort Underestimated Duration Underestimated Effort Overestimated Duration Overestimated Effort Overestimated Duration Underestimated

15. smi COCOMO Forum - October 200415 Observations on New Data The estimation error of the new datasets lie outside the Post-1989 (Cal 2000) dataset error range When each dataset is given its own (local) calibration constant, A, accuracy improves There have been some suggestions on modifying the COCOMO II model –“Globbing” data by application domain or platform and provide different model constants for each “glob” –Add a Cost Driver that accounts for “spread” of data

16. smi COCOMO Forum - October 200416 Proposed New Driver Domain Expertise Driver Definition: –Cumulative knowledge and experience that has been acquired through a thorough track record that comes to represent the core competencies of an organization

18. smi COCOMO Forum - October 200418 Next Steps Finish Early COCOTS calibration –Tailoring and Glue Code activities to analyze –Model definition manual and tool Finish COCOMO II calibration –Consider “Globbing” over adding a new driver Start COCOMO II Driver Elaboration –Make some driver descriptions less subjective –Crisper definitions

19. smi COCOMO Forum - October 200419 For more information, requests or questions Brad Clark Software Metrics, Inc. brad@software-metrics.com Ye Yang USC-CSE yey@usc.edu