1. (Software Engineering group, IDI, NTNU):
Reidar Conradi et al.
(Software Engineering group, IDI, NTNU):
A meta-meta study of Systematic Literature Reviews (SLRs) in Evidence-Based Software Engineering (EBSE)
SU meeting, 11 Sept. 2009
Reidar Conradi, 24,jan.08

2. Table of Contents
Eternal Discussion: Relevance vs Rigor (Robert Glass: Software Creativity, book 1995)
From ESE to EBSE?
SLRs: the main EBSE method – is it delivering? Cost/benefits?
Proposed empirical study of industrial relevance of SLRs: European co-effort with 5? industrial panels

3. Qualitative vs Quantitative methods
Research papers along CS - SE - IS axis:
SE w/ lowest share empirical validation,16%.
Research methods:
Qualitative for relevance: interpreted and subjective context gives meaning, - ex. Observation, Grounded theory.
Quantitive for rigor: “neutral” objects, how to “control” the context – ex. Surveys.
Hybrid: - ex. action research, triangulation. 3

4. ESE vs EBSE 1985: Empirical SE (ESE), Basili et al.
2004: Evidence-based SE (EBSE), coined by Kitchenham et al. after similar E-B Medicine (EBM):
Cochrane centers, each w/ a delegated subdomain; clinical trials w/ s persons
Emphasis on SLRs and E-B guidelines for “best practice”
But humans more than “DNA-driven robots”; cognition w/ own will and initiative, skill factors, reflection, understanding, learning 4

5. SLR protcol, overall
P1: Common guidelines for how to conduct empirical studies,
P2: Common rules for how to report the findings in ensuing papers,
P3: Protocols for how to perform and document systematic reviews of empirical papers in a domain,
P4: Make a complete technical report or a shorter publishable paper about the findings from P3, both to SE researchers and practitioners.
(P5: disseminate these reports and papers.) 5

6. SLR protocol, high-level reviewing
P3.1 Establish a set of “publication channels”, our search universe with relevant journal issues and conf. proceedings (N=25000);
P3.2 Check textual terms: Use Internet search engines on the paper titles, abstracts and keywords of the papers in these channels to select initially promising papers, in two iterations and guided by tailored textual AND/OR queries (N=4000);
P3.3 Check paper abstracts: Let two independent researchers read and agree upon the remaining abstracts and eventually acquire the underlying papers (N=1000);
P3.4 Check paper contents: Do similar tightening of the full textual papers, using the same two persons and applying predefined and formalized acceptance criteria (purely quantitative), until we have a stable set of relevant and sound papers (N= ):
All this requires a gargantuan effort, and any automation is appreciated (see ??). 6

7. Ex. Some SLR findings, 1
Ex. Review of papers on Controlled Experiments:109 experiments in 102 papers. 19 repeated experiments, and 90% of local ones with same result, but only 10% of the foreign ones with same result.
Ex. Review of papers on Cost Estimation: Consistent 30% cost/schedule overruns. Humans with domain insight estimate better than formal model 7

8. Ex. Some SLR findings, 2
Ex. Review of papers on Agile Development: Inconclusive results on Scrum.
Ex. Review of papers on Pair Programming: Gives lower defects at higher cost – obvious!
Ex. Review of papers on Software Maintenance: ??
Ex. [Dybå et al. 2006] Statistical Power: Must have bigger sample sizes in experiments.
Ex. Theory Use: too little and too fragmented.
Ex. Knowledge Management (tools): No information on actual use or success factors. 8

9. Ex. Some SLR findings, 3
Ex. Defect detection: Inspections best for designs.
Ex. Unit testing: Proposing a conceptual framework.
Ex. Own Motivation 9

10. 10 main project kill factors:
Goals, commitment.
Project management.
Communication.
Technical personnel.
Estimation, tracking.
Bad requirements.
Changing requirements.
Product quality.
Development technology.
Software reuse. 10

11. Proposed research: Check industrial relevance of SLR findings
Pick SLRs and 5 researcher colleagues
Make abstracts of these, precise and unbiased!
Make research questions and related evaluation criteria/metrics/interview-guide
Try out eval.criteria, check validity
Select 5 industrial panels in 5 countries
Read and rate the SLRs, done in interviews by int’l colleagues
Sum-up 11

12. Possible research questions
RQ1: Is there an appropriate balance between rigor and relevance? Answer: let professionals read condensed reviews, and propose ordinal marks on both properties.
RQ2: Is there a simple way to assess the cost/benefits of the reviewing process? Answer: Try to record some previous effort estimates for paper selection and processing.
RQ3: How do informal lifecycle models (agile or Open Source ones) interfere with EBSE? Answer: ask what ongoing measurements are being recorded in such projects.
RQ4: Is there any systematic context management to ensure external validity? Answer: check how artifacts and subjects are being described. 12

13. Possible SLR evalution, part1
Ex. Format of SLR presentation
Theme, Title, Authors and their affiliation.
General summary: 4-5 text lines, as prose text.
Concrete summary: 5-10 bullet lines of advice, guidelines, …
(Max one page; give one example in this paper?)
Ex. Personalia for each acad./industr. responent
Respondent data: name, background (education, job experience), personal SE interests.
Knowledge/interests/background of EBSE etc.?
Which review paper to summarize (theme and ID)? 13

14. Possible SLR evaluation, part2
Which SLR to evaluate (subtopic and ID)?
Review rating (ternary scale , with 1.0 as best):
General theme relevance for own job?
Concrete relevance and substance of SLR?
Newness of contents (“aha” factor).
Perceived validity and rigor of SLR
Try to construct an overall mark (0..4) for all this?
Proposed changes in summary format?
Proposed changes for fill-in procedure? 14

15. Conclusion Still hunting for the needed 25-30 SLRs
Rethinking the whole research process
Recruiting co-researchers with industrial panels
Sweating! 15