ESCOM/April 2001Aristotle University/Singular Int’l1 BRACE: BootstRap based Analogy Cost Estimation Automated support for an enhanced effort prediction method I. Stamelos, L. Angelis Aristotle Univ. Thessaloniki E. Sakellaris Singular International

ESCOM/April 2001Aristotle University/Singular Int’l2 Cost Estimation Methods Expert judgement (experience – based estimation) Algorithmic cost estimation (statistical models such as regression) Estimation by analogy (Case Based Reasoning-comparison)

ESCOM/April 2001Aristotle University/Singular Int’l3 Estimation by Analogy (EbA) Characterise new project by certain attributes and place it into a historical data set

ESCOM/April 2001Aristotle University/Singular Int’l4 Estimation by Analogy (cont’d) Calculate distances of the new project from the completed ones and find few "neighbour" projects. Estimate the unknown effort by a location statistic (mean, median) of the efforts of the neighbour projects

ESCOM/April 2001Aristotle University/Singular Int’l5 Distance Metrics

ESCOM/April 2001Aristotle University/Singular Int’l6 Efficiency of EbA Shepperd and Schofield (1997): Superiority of EbA when compared to OLS regression models with 9 industrial datasets (Angel Tool used for EbA calibration) Other researchers (Myrtveit-Stensrud, Briand et al., Jeffery et al.) have recently observed contradicting results

ESCOM/April 2001Aristotle University/Singular Int’l7 Research Issues in EbA Method Accuracy –Extend calibration options: choice of distance metric –Application of EbA with proper historical data Generation of Interval Estimates –Calculation of Confidence Intervals (%CI) –Other measures of accuracy (bias, etc) Angelis, Stamelos, ‘A Simulation Tool for...’, EMSE, March 2000

ESCOM/April 2001Aristotle University/Singular Int’l8 Bootstrap Non parametric bootstrap: –Draw with replacement from the sample, a large number of new samples (of same size) –Estimate each time the effort of the new project –Use the empirical distribution (or an estimation) of the bootstrap samples in order to obtain confidence intervals Parametric bootstrap: based on the multivariate distribution of the original dataset

ESCOM/April 2001Aristotle University/Singular Int’l9 EbA calibration with Bootstrap MMRE - PRED(25) Distributions (Albrecht data set)

ESCOM/April 2001Aristotle University/Singular Int’l10 Confidence Interval Estimation with Bootstrap

ESCOM/April 2001Aristotle University/Singular Int’l11 Comparison of EbA and Regression Confidence Intervals (Abran-Robilland Data Set)

ESCOM/April 2001Aristotle University/Singular Int’l12 BRACE Functions Definition of attributes and project characterization Project/attribute management (e.g. exclusion of projects/attributes from calculations) Calibration of EbA (with and without bootstrap) including the various distance metric options Generation of estimations for a single project (with and without bootstrap) Typical utility functions and file management facilities

ESCOM/April 2001Aristotle University/Singular Int’l13

ESCOM/April 2001Aristotle University/Singular Int’l14

ESCOM/April 2001Aristotle University/Singular Int’l15 A case-study on software projects for the industry The ISBSG Cost Data Base  International Software Benchmarking Standards Group (Australia)  Non profit organization collecting software project data from around the world  Release 6 contains 789 software projects from 20 countries

ESCOM/April 2001Aristotle University/Singular Int’l16 ISBSG Project Data Project Nature (Organisation Type, Business Area Type, Application Type, …) Project Work Effort Data (man-hours) Project Size Data (Function Points) Project Quality Data (defects)...

ESCOM/April 2001Aristotle University/Singular Int’l17 Supply Chain ISBSG Project Subset  59 projects implementing information systems for manufacturing, logistics, warehouse management, …  characterized through effort, size, elapsed time, team size, project nature attributes  accurate project attribute measurement  average productivity ~ 190 FP/ 1000 mh

ESCOM/April 2001Aristotle University/Singular Int’l18 BRACE Application  Various strategies were tried because of missing values in project characterisation  Best strategy pursued a trade-off between number of projects and attributes  Precision was measured through project jackniving  Different treatment for elapsed time and max team size

ESCOM/April 2001Aristotle University/Singular Int’l19 EbA Precision Results Best parameter configuration: 30 projects, Canberra distance, one analogy, size adjustment: MMRE = 28.84%, PRED(25) = 46.67% When using elapsed time and team size Minkowski, λ=3 distance MMRE = 23.84%, PRED(25) = 70.37%

ESCOM/April 2001Aristotle University/Singular Int’l20 Future work  Project portfolio estimation  Clustering of the cost dataset  Implementation of Parametric Bootstrap  Optimisation techniques in calibration  Replication of the study with new ISBSG release (~1000 projects)