1. LITERATURE REVIEW

2. SYSTEMATIC LITERATURE REVIEW
WHAT IT IS?
WHY IT IS?
HOW TO CONDUCT IT?
WHAT IS THE DIFFERENCE?

3. WHAT IS SYSTEMATIC LITERATURE REVIEW (SLR)?
Often refer as Systematic Review (SR)
Definition:
A rigorous summary of all primary research evidence relevant to a focus questions.
Reviewing evidence and synthesising findings.
A review of the research literature using systematic and explicit accountable methods.

4. WHAT IS SYSTEMATIC LITERATURE REVIEW (SLR)?
Often refer as Systematic Review (SR)
Definition:
A rigorous summary of all primary research evidence relevant to a focus questions.
Reviewing evidence and synthesising findings.
A review of the research literature using systematic and explicit accountable methods.
Tight, detail
Specific questions
Creating something new from separate element
Critical appraisal & analysis
Undertaken according to a fixed plan or system or method
Clear, understandable statement of all the relevant details

5. WHAT IS SYSTEMATIC LITERATURE REVIEW (SLR)?
Reviewing research systematically involves THREE key activities:
Identifying and describing the relevant research (‘mapping’ the research).
Critically appraising research reports in a systematic manner.
Bringing together the findings into a coherent statement (known as synthesis)

6. MOTIVATIONS TO DO SLR
To systematically accumulate, organize, evaluate, and synthesize all existing research evidence of your research area.
To present fair evaluation of a research topic by using a trustworthy, rigorous, and auditable methodology.
To produce reliable and unbiased results.
To identify gaps in the existing research that will lead to topics for further investigation.
To provide as a background to position new research activities

7. The systematic review process
Search bibliographic databases
Formulate research questions
Design search strategy
Further selection of primary studies using inclusion criteria
Identify possible papers from titles/abstracts
Retrieve papers
Quality appraisal
Synthesis
Extract data
Formulate research / policy conclusions

8. STAGES OF SYSTEMATIC REVIEW
Report of findings
Conclusions are made based on data
Data are systematically pooled
Information from studies is critically appraised
Relevant studies are highlighted by search strategy
Inclusion & Exclusion criteria are specified
Clear RQ(s) is defined

9. STEPS IN CONDUCTING SLR
Planning Phase
Formulate the review’s research questions
Develop the review’s protocol
Conducting Phase
Search the relevant literature
Perform selection of primary studies
Perform data collections
Assess study’s quality
Conduct synthesis of evidence
Reporting Phase
Write up the SLR report/paper

10. PLANNING PHASE Formulate the review’s research questions
Develop the review’s protocol

11. PHASE 1 STEP 1: FORMULATE THE REVIEW’S RESEARCH QUESTIONS

12. The choice of review question depends on:
Who is prepared to fund a review or reviews.
Example: Ministry of Health funded the study on Zika outbreak
Who is prepared to invest their time in conducting a review
What questions these people (conducting the review) see as important
Is the most important part in any SLR
Is not necessarily the same as question(s) addressed in your research.
Is used to guide the search process
Is used to guide the extraction process

13. FORMULATING RQ: Petticrew & Robert (2006) suggest that the formulation of RQs about effectiveness of a treatment should focus on 5 elements known as PICOC

14. Example: Title: “Cross verses Within-Company Estimation Cost Estimation Studies: A Systematic Review”

15. Example: Title: “Empirical Studies of Pair Programming for CS/SE Teaching in Higher Education: A Systematic Literature Review”

16. PHASE 1 STEP 2: DEVELOP THE REVIEW’S PROTOCOL

17. Protocol: A plan that specifies the basic review procedures.
Components of a protocol:
Background – rationale for survey
RQ – define before starts the survey/review
Search terms – terms/keywords, identify sources (resources, databases, journals, conferences, report etc)
Selection criteria – inclusion and exclusion criteria
Quality checklist and Procedures - Criteria used to evaluate quality of primary sources
Data extraction strategy
What data will be extracted from each primary source
How to handle missing information
How data reliability will be addressed - Usually multiple reviewers
Where data will be stored
Procedure for data synthesis – format for summarising the data
Project schedule – log information of data/reference retrieval

18. CONDUCTING PHASE Search the relevant literature
Perform selection of primary studies
Perform data collections
Assess study’s quality
Data Extraction
Conduct synthesis of evidence

19. PHASE 2 STEP 1: SEARCH THE RELEVANT LITERATURE

20. Involves a comprehensive and exhaustive searching of studies to be included in the review.
Search strategies are usually iterative and benefit from:
Preliminary searches (to identify existing review and volume of studies)
Trial searches (combination of terms from RQ)
Check the search results against list of known studies
Consult the experts in the field
Identify source of evidence

21. Source of evidence:
Digital libraries. Examples: INSPEC , El Compendex, Science Direct, Web of Science, IEEExplore, ACM Digital library
Reference lists from relevant primary studies and review articles
Journals (including company journals such as the IBM Journal of Research and Development)
Grey literature (i.e. technical reports, work in progress)
Conference proceedings
Research registers. Examples: LinkedIn, research gate
The Internet. Example: google scholar, cite seer, Agile Alliance
Direct contact specific researcher(s)

22. Documenting the source:
The process of conducting SLR must be transparent and replicable.
The review should be documented in sufficient detail
The search should be documented and changes noted.
Unfiltered search results should be saved for possible reanalysis

23. PHASE 2 STEP 2: Perform selection of primary studies

24. Primary studies need to be assessed for their actual relevance
Set the criteria for including or excluding studies
Inclusion & exclusion criteria should be based on RQ
Study selection is a multistage process.

25. EXAMPLE:
Kitchenham et al. (2007) work : “Cross verses Within-Company Estimation Cost Estimation Studies: A Systematic Review”

26. EXAMPLE:
Salleh et al. (2011) work: “Empirical Studies of Pair Programming for CS/SE Teaching in Higher Education: A Systematic Literature Review”

27. PHASE 2 STEP 3: Perform data collections

28. Collect the literature from the identified source
Read, Summarize
Keep and save: Tools for Reference Management

29. PHASE 2 STEP 4: Assess study’s quality

30. Reasons: To provide more detailed Inclusion/Exclusion criteria
To check whether quality differences provide an explanation for differences in study results
As a means of weighting the importance of individual studies when results are being synthesized.
To guide the interpretation of findings and determine the strength of inferences.
To guide recommendations for further research.

31. Example:
Salleh et al. (2011) work: “Empirical Studies of Pair Programming for CS/SE Teaching in Higher Education: A Systematic Literature Review”

32. PHASE 2 STEP 5: Data Extraction

33. Involve reading the full text article.
Data extracted from primary studies should be recorded using data extraction form.
Collect all the information that can be used to answer the RQ and the study’s quality criteria.
Both quality checklist and review data can be included in the same form.
In case of duplicates publications (reporting the same data), refer to the most complete one.
For validation, a set of papers should be reviewed by 2 or more researchers. Compare results and resolve any conflicts.

34. PHASE 2 STEP 6: Conduct synthesis of evidence

35. Key objectives of data synthesis (Cruzes & Dyba, 2011):
Involves collating and summarizing the results of the included primary studies
Key objectives of data synthesis (Cruzes & Dyba, 2011):
to analyse and evaluate multiple studies
to select appropriate methods for integrating or providing new interpretive explanations about them

36. REFERENCES
Budgen,D., Charters, S., Turner, M.,, Brereton, P., Kitchenham, B., Linkman, S. (2006) Investigating the applicability of the evidence-based paradigm to software engineering, Proc Int’l Workshop on interdisciplinary software engineering research, pp. 7 – 13
Cruzes, D.S., and Dyba, T. (2011) Research Synthesis in Software Engineering: A tertiary study, Information and Software Technology, 53(5), pp
Jalali, S. & Wohlin, C. (2012). Systematic Literature Studies: Database Searches vs.Backward Snowballing, Proceedings of the 6th Int'l Symp. Empirical Software Engineering & Measurement (ESEM 2012).
Kampenes, V. B., Dyba, T., Hannay, J. E., & Sjoberg, D. I. K. (2007). A systematic review of effect size in software engineering. Information and Software Technology, 49,
Salleh, N., Mendes, E., & Grundy, J. (2011). Empirical studies of pair programming for CS/SE teaching in higher education: A systematic literature review. IEEE Transactions on Software Engineering, 37(4): 509 –
Kitchenham, B., Mendes, E., Travassos, G.H. (2007) A Systematic Review of Cross- vs. Within- Company Cost Estimation Studies, IEEE Transactions on Software Engineering, 33 (5), pp

37. - Thank you: PM Dr. Zuraini & Dr. Rashimah
SLR PRACTICAL
- Thank you: PM Dr. Zuraini & Dr. Rashimah

38. SLR RQ: What are the factors that influence acceptance on augmented reality technology?

39. DATABASE

41. SEARCH RELEVANT STUDIES

44. DATA COLLECTION

48. REPORTING PHASE
Write up the SLR report/paper

49. REPORT STRUCTURE Introduction
General introduction about the research. State the purpose of the review. Emphasize the reason(s) why the RQ is important. State the significance of the review work and how the project contributes.
Main Body
Review method : briefly describe steps taken to conduct the review
Results: Findings from the review, i.e. the synthesis
Discussion: Implication of review for research & practice, Threats to Validity
Conclusions

50. SAMPLE OF SLR REPORT
Rouhani, Babak Darvish, et al. "A systematic literature review on Enterprise Architecture Implementation Methodologies." Information and Software Technology 62 (2015): 1-20.
Salahuddin, Lizawati, and Zuraini Ismail. "Classification of antecedents towards safety use of health information technology: A systematic review." International Journal of Medical Informatics (2015):