1. Introduction to Systematic Reviews
Dawn Craig
Institute of Health & Society
@dawn_craig

2. Aims of this session…. Outline what a systematic review is
To discuss scope and the formulation of a review question
The purpose and format of a protocol
Systematic searching/screening of studies
Data extraction/quality appraisal and intro to evidence synthesis

3. What is a systematic review?
SYSTEMATIC: Done or acting according to a fixed plan or system: methodical
REVIEW: A critical appraisal of a book, play or other work

4. What is a systematic review?
“A systematic review is a review in which there is a comprehensive search for relevant studies on a specific topic, and those identified are then appraised and synthesized according to a predetermined and explicit method.”* (*Klassen et al. Guides for reading and interpreting systematic reviews. Arch Pediatr Adolesc Med 1998;152: )
A systematic review attempts to collate all empirical evidence that fits pre- specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman 1992, Oxman 1993)

5. Why we need systematic reviews
Minimise the impact of bias/errors
Can help to end confusion
Highlight where there is not sufficient evidence
Combining findings from different studies can highlight new findings
Can mitigate the need for further trials

6. Why we need systematic reviews
Facilitate rational decision making
Health care providers, researchers and policy makers are inundated with unmanageable amounts of information
Over 20 million citations in PubMed
Approx. 75 to 100 RCTs published daily
Usually impossible to consider all relevant individual primary research studies in a decision making context
Enable practitioners to keep up to date with evidence accumulating in field and to practice evidence-based medicine

7. Why not traditional reviews
‘Unscientific’ rarely pre-specify or make methods explicit
Rarely transparent or reproducible
Usually qualitative, subjective, opinions of individual
Often incomplete, filing cabinet or MEDLINE review
Difficult to make sense across groups of studies, especially when conflicting based on qualitative reading alone

8. Hierarchy of evidence

9. Who undertakes systematic reviews?
Cochrane/Campbell Collaboration
NICE/Regulatory bodies
Health Technology Assessment
Academics/researchers/Clinicians
MSc/PhD students

10. Who undertakes systematic reviews?
Multidisciplinary teams
Clinicians
Health services researchers
Information scientists
Statisticians
Health Economists
Patient and public involvement – particularly for guidelines

11. Key Stages in a Systematic Review- the process
Define research/review question
In consultation/collaboration with the clinical community, commissioners and patient/public representatives
Identify relevant studies
Develop a comprehensive search strategy and undertake systematic searches of the literature
Assess eligibility
Select those studies which meet the pre-defined inclusion criteria
Data extraction /checking
Develop data extraction from into which study information and outcome data can be extracted, checked & verified
Synthesis
Narratively and/or statistically summarise/describe the data, exploring similarities and differences between studies.
Develop review protocol
Pre-specify the type of studies to be included, the methods of collating, appraising and analysing data
Knowledge translation
Review details and results are disseminated to relevant target audiences using appropriate formats
Study assessment/appraisal
Assess the quality and validity of the included studies using the pre-defined method.

12. Define research/review question
Questions may be broad or narrow
Well-formulated questions will guide many aspects of the review process
Searching strategy
Inclusion/exclusion criteria
Data extraction
Choice of synthesis method
Presentation/dissemination of findings

13. Quick Activity
Discuss a very broad question and how you might narrow it? (10 mins)
Discuss the potential limitations of your review questions
If time and resource were not a limitation – consider how useful would the answer to your review question be?

14. Current guidance
a clear and concise statement of a review's objectives (or questions) is critical and should begin with a precise statement of the primary objective, including the interventions reviewed and the targeted problem; ideally, this would be presented in a single sentence
Cochrane & Prisma Statment

15. Current guidance
“To assess the effects of [intervention or comparison] for [health problem] in [types of people, disease or problem, and setting if specified].”
Several criteria/frameworks proposed to help guide question development

16. Determining the scope is a decision dependent upon multiple factors:
Question formulation
Determining the scope is a decision dependent upon multiple factors:
Perspectives regarding a question’s relevance and potential impact;
Supporting theoretical, biologic and epidemiological information;
The potential generalizability and validity of answers to the questions;
Available resources;
The wider literature base – has a recent high-quality SR been conducted?

17. Question formulation
Advantages and disadvantages to both broad and narrow questions
The validity of very broad question may be criticized for ‘mixing apples and pears’; but advantages might include
Comprehensive summary of the evidence
Generalizability of findings
Most obvious advantage of narrow focus is clarity of objectives and ease of reading; but disadvantages might include
Sparse evidence may limit findings/usefulness

18. Question formulation Often dealing with complex interventions
Might be a need to develop working definitions of the intervention of interest
Several options on how to do this (pragmatic real world v theoretical, logic models, etc.)
Use content experts outside the review team to ensure that the resulting definitions are likely to be robust and meaningful

19. A protocol is an essential component of the systematic review process
Protocol Development
A protocol is an essential component of the systematic review process
Helps to ensure careful a priori planning
Consistency
Transparency
Integrity
Integral part of the process for leading organisations/publication process

20. Protocol Development
One of the features that distinguish a systematic review from a narrative review is the pre-specification of criteria
Inclusion
Exclusion
Methods
Outcomes to be synthesised
Etc.

21. PROSPERO – CRD initiative
Search for existing current reviews
Register their planned review online
Publish protocol online
Update record on Prospero website as the review progresses
Avoids duplication of reviews

22. Searching for Information
Types of
Studies (RCTs, non-RCTs, cohort/case-controlled)
Population and setting
Interventions
Outcome measures
Cochrane Handbook and CRD Guidelines
Both provide explanations re the difference study designs, likely biases and issues to consider when including them

23. Searching for Information
MESH terms and key words/synonyms
Medical Subject Heading – controlled vocabulary thesaurus used for indexing articles
young; adoles*; teen*; child*
*end of the ‘stem’ of the word it will automatically search for all the endings for that word stem
Child* will also return children, childbearing, childbirth and so on…

24. Searching for Information
Word variants
AIDS
acquired immunodeficiency syndrome
acquired immuno-deficiency syndrome
acquired immune deficiency syndrome
acquired immune-deficiency syndrome
Synonyms e.g. Newborn: infant, toddler, baby, etc.
Plurals e.g. child : children OR teenager : teenagers
Spelling variants (UK vs US) e.g. randomise/randomize

25. Searching for Information
Where to search
Electronic databases: Medline, Embase, Cochrane, PsycInfo, etc.
Grey literature, dissertations, theses, conference proceedings, national bodies (NICE, HTA), clinical trial database (
Look at the databases own guidance for searching they vary!

26. APHO staff conference 2009
Boolean operators 26

27. Searches in medline 2006-2010 database
Alcohol – subject (all subheadings)
5565
Alcohol – subject or keyword
12052
([young or adoles* or teen*] [review or system*]
19180
Rows above combined with or
6791

28. More specific – get help!

29. Selection of Studies Reference manager software package
Endnote – RefMan – ProCite – Mendeley
Import results and screen
Assess titles/abstracts against your predetermined criteria
If in doubt include
Retrieve full text articles of initial selections
Assess full text for inclusion
Requires judgement (>1 reviewer)
Check reviewer agreement (3rd review to resolve)
Use a selection form to ensure consistency and record decisions

30. Data Extraction/Quality Appraisal

31. Data Extraction
Be clear what information you want from the studies:
Study details
Data for your analysis
Information will need to be collected relating to:
Methodology
Population
Interventions being compared
Outcomes evaluated

32. Give consideration to….
What effect measures you are you going to calculate
What data do you need to do this?
How are you planning to group studies for the analysis?
By intervention?
By study design?
What information do you need to extract to enable you to organise and analyse the way you want?
REMEMBER YOUR PROTOCOL – IT IS YOUR ROADMAP, FOLLOW IT!

33. Level of judgement is required
How much to extract??
Level of judgement is required
Sufficient to describe studies
Sufficient to allow you to undertake the planned analysis
Sufficient so you do not need to return to the full text papers
However
You need to limit unnecessary detail

34. Data extraction software?
There is a wide selection of software to choose from
Selection depends on a number of factors
Main considerations are probably
What are you are familiar with?
What package best suits your data?
How many included studies do you have?

35. Which software?
Word
Excel
Access
EPPI reviewer
COEVIDENCE
REVMAN
????

36. Consistency/Standardisation
We all have to be doing the same thing
Essential >one reviewer is extracting data
Data must be interpreted in the same way by all reviewers
Independent piloting of data extraction forms – always one standardised form
Regular discussion of progress/disagreements
Regular comparison of data extraction – don’t wait till the end

37. Efficient data extraction
Once data extraction is complete you may need to:
Sort/search your data
Filter data
Calculate frequencies
Transform data (e.g. SE to SD)
Categorising/coding data will make these tasks easier
Needs to be implemented with consistency by the whole team
A database can be designed to have this functionality

38. Things to consider Are you including more than one study design?
You may need separate forms for each study design
However, you are still answering the same question, so make sure the core information extracted is the same
Have one or a few studies reported data differently from the others?
Will the data still be useful?
Should you include it?
Make sure the core information extracted is the same
You may need to update the form, or have more than one form
Any changes need to be agreed and made consistently

39. Stay on track…… Be careful about collecting ‘extra’ data
It is very tempting to collect data that are not directly relevant to the review question
The data needed to answer the review question should have already been decided (REMEMBER YOUR PROTOCOL)
Collect data for good reasons – stay focused and don’t get side-tracked
Time and effort to collect, only to find it is not useful

40. Quality Assessment & Critical Appraisal
Why bother????
What are we trying to achieve?
Not all published and unpublished literature is rigorous!
being in a journal doesn’t mean it is good
Quality may be used as an explanation for differences in study results or to guide interpretation of findings, strength of inferences

41. Quality Assessment & Critical Appraisal
Quantitative studies
Internal validity
Bias: selection; performance; detection; attrition; reporting
External validity
Move away from checklists/numerical scores to domain based assessment
Cochrane Risk of Bias - RCTs
QUADAS 2 – diagnostic accuracy
ROBIS for systematic reviews

42. Quality Assessment & Critical Appraisal
Qualitative studies
Three broad categories
Rigour: has a thorough and appropriate approach been applied to key research methods in the study?
Credibility: are the findings well presented and meaningful?
Relevance: how useful are the findings to you and your organisation?

43. CASP appraisal checklist
Clear aims of research (goals, why it is important, relevance)
Appropriate methodology
Sampling strategy
Data collection
Relationship between researcher and participants
Ethical issues
Data analysis
8. Findings
9. Value of research (context dependent)

44. Data Synthesis
Building up; putting together; making a whole out of the parts; the combination of separate elements of thought into a whole; reasoning from principles to a conclusion

45. Results from different studies need to be synthesised
Data Synthesis
Results from different studies need to be synthesised
Are studies and results similar enough to be combined into a single numerical result?
NO – qualitative descriptive/narrative summary
YES – quantitative meta-analysis
Heterogeneity
Difference in results can arise due to differences in study design, population, selection, intervention delivery
How similar is similar? Results from heterogeneous studies should not be pooled

46. Instead of/alongside meta-analysis Potential bias in presentation
Narrative synthesis
Instead of/alongside meta-analysis
Potential bias in presentation
Lack of a take home message

47. Tools for narrative synthesis
Partly informed by methodological work in qualitative synthesis
Tabulation
Groupings and clusters
Vote counting as a descriptive tool
Examination of moderator variables (elements of e.g. setting, population)
Rodgers et al Evaluation

48. Meta-analysis/forest Plot

49. Most important thing: Be organised!!!

50. Gantt charts

51. Use a reference manger to sift and store Keep all citations retrieved
Reference management
Use a reference manger to sift and store
Keep all citations retrieved
Add in those you can’t download
Use to de-duplicate results
Sift citations for inclusion/exclusion
Can use codes/notes

52. Avoid using draft, draft 1, final final, etc. Clear naming convention
Version control
Dates – YYYYMMDD
Version numbering
v0.1 = first draft
v1.0 = final version
v1.1 = minor amendments to final version
v2.0 = major revision
Avoid using draft, draft 1, final final, etc.
Clear naming convention
E.g. Date_project_title
20/11_Autism HTA_resultsv1.2.doc

53. FMS Systematic review group
Other help available
FMS Systematic review group
Informal monthly session where methods are discussed and issues can be raised or
alternatively you have my contact details
MSc in Public Health and Health Services Research (~October 2016)
10 credit module ‘Introduction to systematic reviewing and critical appraisal’

54. Thanks for your attention!
Contact details:
Dawn Craig
Institute of Health & Society
@dawn_craig