1. Evidence-Based Software Engineering and Systematic Reviews
Barbara Kitchenham

2. Agenda The evidence-based paradigm
Evidence-Based Software Engineering (EBSE)
Systematic Reviews

3. The Evidence-Based Paradigm
Evidence-based medicine has changed research practices
Medical researchers found
Failure to organise existing medical research cost lives
Clinical judgement of experts worse than systematic reviews
Evidence-based paradigm adopted by many other disciplines providing service to public
Social policy
Education
Psychiatry

4. Goal of EBSE
EBM: Integration of best research evidence with clinical expertise and patient values
EBSE: Adapted from Evidence-Based Medicine
To provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision making process regarding the development and maintenance of software
Anticipated benefits
Common goals for research groups
Help for practitioners adopting new technologies
Means to improve dependability
Increase acceptability of software-intensive systems
Input to certification process

5. Practicing EBSE Convert information need into answerable question
Track down best evidence
Critically appraise evidence
Integrate critical appraisal with SE expertise and stakeholder requirements
Evaluate and improve above steps

6. Systematic Reviews - 1/2 A systematic review is
An overview of research studies that uses explicit and reproducible methods
Systematic reviews aim to synthesise existing research
Fairly (without bias)
Rigorously (according to a defined procedure)
Openly (ensuring that the review procedure is visible to other researchers)

7. Systematic Reviews – 2/2 Support Evidence-based paradigm
Start from a well-defined question
Step 1
Define a repeatable strategy for searching the literature
Step 2
Critically assess relevant literature
Step 3
Synthesise literature
Step 4 (but only partially)

8. Advantages
Provide information about effects of a phenomenon across wide range of settings
Essential for SE where we have sampling problems
Consistent results provide evidence that phenomena are
Robust
Transferable
Inconsistent results
Allow sources of variation to be studied
Meta-analysis possible for quantitative studies

9. Anticipated Benefits Create a firm foundation for future research
Position your own research in the context of existing research
Close areas where no further research is necessary
Uncover areas where research is necessary
Help the development of new theories
Identify common underlying trends
Identify explanations for conflicting results
Should be a standard research methodology

10. Disadvantages Require more effort than informal reviews
Difficult for lone researchers
Standards require two researchers
Minimising individual bias
Incompatible with requirements for short papers

11. Value of Systematic Reviews
Can contradict “common knowledge”
Jørgensen and Moløkken reviewed surveys of project overruns
Standish CHAOS report is out of step with other research
May have used inappropriate methodology
Jørgensen reviewed evidence about expert opinion estimates
No consistent support for view that models are better than human estimators

12. Systematic Review Process
Develop Review Protocol
Plan Review
Validate Review Protocol
Identify Relevant Research
Select Primary Studies
Conduct Review
Assess Study Quality
Extract Required Data
Synthesise Data
Write Review Report
Validate Report
Document Review

13. Developing the Protocol
Review protocol
Specifies methods to be used for a systematic review
Predefined protocol
Reduces researcher bias by reducing opportunity for
Selection of papers driven by researcher expectations
Changing the research question to fit the results of the searches
Good practice for any empirical study

14. Protocol Contents -1/2 Background Research question
Rationale for survey
Research question
Critical to define this before starting the research
Strategy used to search for primary sources
Individual studies of the phenomenon of interest

15. Protocol Contents – 2/2 Strategy to find primary studies
Search terms, resources, databases, journals, conferences
Procedures for storing references
How publication bias will be handled
Grey literature
Direct approach to active researchers
How completeness will be determined
Useful to have the baseline paper to set start date
Selection Strategy
Inclusion/exclusion criteria
Handling multiple papers on one experiment

16. Protocol Contents- 2/3 Quality assessment criteria Data extraction
Criteria used to evaluate quality of primary sources
Data extraction
What data will be extracted from each primary source
How to handle missing information
How data reliability will be addressed
Usually multiple reviewers
Where data will be stored
Procedures for data synthesis
Formats for summarising data
Measures and analysis if meta-analysis is proposed
Should tested during protocol construction

17. Research Question – 1/2 Question types for EBSE
Assessing the effect of an SE technology
Assessing the frequency or rate of a project development factor
E.g. Rate of project failures
Identifying cost and risk factors
Identifying impact of technology on reliability, performance, cost
Possible to have more general questions for other purposes
Review of research in software engineering (Glass, et al., 2002)

18. Research Question – 2/2 Question structure Population Intervention
People, projects types, applications types affected by the intervention
Intervention
Software method, tool, procedure
Outcomes
Impact of technology in terms relevant to practitioners
Cost, quality, time to market
Experimental designs
Any constraints on type of primary studies to be included

19. Next steps are easy!? Conduct the review Document the Review
Enact the protocol
Expect further iterations of
Search strategy
Selection criteria
Data extraction
Record any deviations from protocol
Document the Review
Using procedures defined in protocol

20. Conclusions Evidence-based approach Systematic reviews
Revolutionised medicine
May be relevant to SE
Systematic reviews
Support the evidence-based approach
Valuable as a research tool
Even if we don’t accept EBSE

21. References
Australian National Health and Medical Research Council. How to review the evidence: systematic identification and review of the scientific literature, IBSN
Australian National Health and Medical Research Council. How to use the evidence: assessment and application of scientific evidence. February 2000, ISBN
Cochrane Collaboration. Cochrane Reviewers’ Handbook. Version December 2003.
Glass, R.L., Vessey, I., Ramesh, V. Research in software engineering: an analysis of the literature. IST 44, 2002, pp
Magne Jørgensen and Kjetil Moløkken. How large are Software Cost Overruns? Critical Comments on the Standish Group’s CHAOS Reports,
Magne Jørgensen. A Review of Studies on Expert Estimation of Software Development Effort. Journal Systems and Software, Vol 70, Issues 1-2, 2004, pp

22. References
Khan, Khalid, S., ter Riet, Gerben., Glanville, Julia., Sowden, Amanda, J. and Kleijnen, Jo. (eds) Undertaking Systematic Review of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition), NHS Centre for Reviews and Dissemination, University of York, IBSN , March 2001.
Kitchenham, Barbara. Procedures for Performing Systematic Reviews, Joint Technical Rreport, Keele University TR/SE-0401 and NICTA T.1, July 2004.
Pai, Madhukar, McCullovch, Michael, Gorman, Jennifer D., Pai, Nitika, Enanoria, Wayne, Kennedy, Gail, Tharyan, Prathap, Colford, John M. Jnr. Systematic reviews and meta-analysis: An illustrated, step-by-step guide. The National medical Journal of India, 17(2) 2004, pp
Sackett, D.L., Straus, S.E., Richardson, W.S., Rosenberg, W., and Haynes, R.B. Evidence-Based Medicine: How to Practice and Teach EBM, Second Edition, Churchill Livingstone: Edinburgh, 2000.