1. 1 A Systematic Review of Cross- vs. Within-Company Cost Estimation Studies Barbara Kitchenham Emilia Mendes Guilherme Travassos

2. 2 Outline Motivation Objective Method –Research Questions, Population, Intervention, Outcome –Search Strategy used for Primary Studies Initial Search Phase Secondary Search Phase –Study Selection Criteria and Procedures for Including and Excluding Primary Studies –Study Quality Assessment Checklists –Data Extraction Strategy Results Conclusions

3. 3 Motivation Contradictory results as to the applicability of cross- company models to estimate effort for single company projects

4. 4 Objective To determine under what conditions individual organisations are able to rely on cross-company-based estimation models

5. 5 Method The first step was to prepare a protocol to be used as a plan for the systematic review. There were no other reviews on the same topic that have been previously conducted.

6. 6 Method: Research Questions Question 1 –What evidence is there that cross- company estimation models are not significantly worse than within- company estimation models for predicting effort for software/Web projects? Question 2 –Do the characteristics of the study data sets and the data analysis methods used in the study affect the outcome of within- and cross- company effort estimation accuracy studies?

7. 7 Method: Research Questions Secondary question: Question 3 –Which estimation method(s) were best for constructing cross- company effort estimation models? Not discussed in the paper due to lack of space

8. 8 Population: cross-company benchmarking data bases of Web and software projects Intervention: effort estimation models constructed from cross-company data, used to predict effort for single company projects Comparison Intervention: effort estimation models constructed from the within- company data only Outcome: the accuracy of the cross- and within-company models Method: Population, Intervention, Outcome

9. 9 Search Strategy used for Primary Studies The search terms used in our Systematic Review were constructed using the following strategy: –Derive major terms from the questions by identifying the population, intervention and outcome ; –Identify alternative spellings and synonyms for major terms. We also included terms identified via consultations with experts in the field and/or subject librarians; –Check the keywords in any relevant papers we already have ; –Use the Boolean OR to incorporate alternative spellings and synonyms ; –Use the Boolean AND to link the major terms from population, intervention and outcome.

10. 10 The complete set of search strings was: (software OR application OR product OR Web OR WWW OR Internet OR World-Wide Web OR project OR development) AND (method OR process OR system OR technique OR methodology OR procedure) AND (cross company OR cross organisation OR cross organization OR cross organizational OR cross organisational OR cross- company OR cross-organisation OR cross-organization OR cross- organizational OR cross-organisational OR multi company OR multi organisation OR multi organization OR multi organizational OR multi organisational OR multi-company OR multi-organisation OR multi-organization OR multi-organizational OR multi- organisational OR multiple company OR multiple organisation OR multiple organization OR multiple organizational OR multiple organisational OR multiple-company OR multiple-organisation OR multiple-organization OR multiple-organizational OR multiple- organisational OR within company OR within organisation OR within organization OR within organizational OR within organisational OR within-company OR within-organisation OR within-organization OR within-organizational OR within- organisational OR single company OR single organisation OR single organization OR single organizational OR single organisational OR single-company OR single-organisation OR single-organization OR single-organizational OR single- organisational OR company-specific) AND (model OR modeling OR modelling) AND (effort OR cost OR resource) AND (estimation OR prediction OR assessment)

11. 11 Initial Search Phase To identify candidate primary sources based on our own knowledge, and searches of electronic databases using the derived search strings

12. 12 Databases/Journals searched Databases –INSPEC –El Compendex –Science Direct –Web of Science –IEEExplore –ACM Digital library Individual journals (J) and conference proceedings (C) –Empirical Software Engineering (J) –Information and Software Technology (J) –Software Process Improvement and Practice (J) –Management Science (J) –International Software Metrics Symposium (C) –International Conference on Software Engineering (C) –Evaluation and Assessment in Software Engineering (manual search) (C) These conferences were chosen as they had previously published primary studies we knew about.

13. 13 Results initial search phase No new relevant papers were found. 772 papers were retrieved, 24 represented the set of 9 known papers. IEEExplore retrieved the largest number of known papers (5) INSPEC, Science Direct and Web of Science each retrieved 4 papers Science Direct, Metrics Proceedings search engine, and Management Science search engine (JSTOR) each missed a known paper that should have been indexed by that search engine. ACM Digital library missed two known papers that should have been indexed by this search engine

14. 14 Secondary search phase To review the references of each of the primary sources identified in the first phase looking for any other candidate primary sources. This process was to be repeated until no further reports/papers seemed relevant. No new references were found. To contact researchers who authored the primary sources in the first phase, or who we believe could be working on the topic. Six researchers were contacted, no one was working on the topic.

15. 15 Study Selection Criteria and Procedures for Including and Excluding Primary Studies Criteria for including a primary study comprised any study that compared predictions of cross-company models with within-company models based on analysis of project data. Criteria for excluding a primary study: –If projects were only collected from a small number of different sources (e.g. 2 or 3 companies) –If models derived from a within-company data set were compared with predictions from a general cost estimation model.

16. 16 Study Quality Assessment Checklists Criteria used to determine the overall quality of the primary studies included six top-level questions and an additional quality issue. Additional quality issue was the size of the within-company data set => Larger WC data sets would lead to more reliable comparisons

17. 17 Data Extraction Strategy For each paper a reviewer was nominated at random as data extractor, checker, or adjudicator. Roles were assigned at random with the following restrictions: –No one should be data extractor on a paper they authored. –All reviewers should have an equal work load (as far as possible).

18. 18 Results: Question 1 [1] [1] Cross-validation for within-company model

19. 19 Results: Question 2 Does quality control on data collection make cross-company models to be as good as within- company models? –S10 contradicts that –Studies S3 and S1 take a different view on quality control (ESA database) –Studies S2 and S6 both agree that stringent quality control is applied to data collection. Problem here is S6

20. 20 Results: Question 2 No consistent evidence that the quality of the studies influences the results –S2 and S3 have lower scores –However S10 has the highest quality score

21. 21 However noticeable difference when we compare the number of projects in the within-company models for S2, S3, S10 (median 63) and S4, S5, S8, S9 (median 10). All the studies where within- company predictions were significantly better than cross- company predictions used small within-company data sets of fair quality. Similar pattern applies to the range of effort values for the entire database Results: Question 2

22. 22 Results: Question 2 No clear patterns were observed for the size metrics used, nor for the procedure used to build the within- company model

23. 23 Meta-analysis? We were unable to carry out meta-analysis on the results because we did not have the residuals from previous studies, or actual and estimated effort. Future publications comparing within- to cross-company predictions are encouraged to also provide residuals for within- and cross-company models, for each single company project.

24. 24 Conclusions The results of our systematic review are inconclusive. Researchers should come to a consensus about the most appropriate experimental protocol for this type of study and use the same protocol for future studies.