1. Systematic and Rapid ReviewS

2. Where did they come from?
Health sector
Research evidence not being included in doctor’s decision-making
Preventable deaths occurred
Spread to other disciplines: education, environmental sciences, engineering, international development
(Antman et al. 1992) The paper showed that traditional review articles and textbooks had often given treatment advice that was dangerously inconsistent with the evidence available at the time they had been written. “Review articles often failed to mention important advances or exhibited delays in recommending effective preventive measures. In some cases, treatments that have no effect on mortality or are potentially harmful continued to be recommended by several clinical experts.”
(IC) was summoned to give evidence to a House of Lords Committee on medical research, and drew on the paper’s findings. The Committee was informed that, five years after a systematic review of controlled trials had shown that thrombolysis reduced the risk of death after myocardial infarction, the Oxford Textbook of Medicine maintained that the benefits of the treatment had not been established (Pentecost 1987) . The following weekend (5 Feb 1995), this contribution to the Committee’s thinking led The Sunday Times to publish an article on its front page under the headline ‘Hundreds killed by doctors relying on outdated manuals

3. Policy Horizons Canada
Why bother?
“Evidence-based, evidence-informed or knowledge-based policy development refers to an approach that levers the best available objective evidence from research to identify and understand issues so that policies can be crafted by decision makers that will deliver desired outcomes effectively, with a minimal margin of error and reduced risk of unintended consequences.”
Policy Horizons Canada

4. What are they? Type of literature review Asks a specific question
Rigorous
Transparent
Replicable
attempt to summarize all pertinent studies on a specific question
can improve the understanding of inconsistencies in diverse evidence
can identify gaps in research evidence to define future research agendas
Question usually about the effectiveness of a program or treatment
Rigorous – very thorough, try to gather all relevant evidence, assess the quality of research
Transparent – stipulate apriori the protocol (state assumptions), recognize methodological biases
Replicable – results can be reproduced

5. What is a systematic review?
A literature review that asks a specific question and has
a clearly stated set of objectives with pre-defined eligibility criteria for studies;
an explicit, reproducible methodology;
a systematic search that attempts to identify all studies that would meet the eligibility criteria;
an assessment of the validity of the findings of the included studies;
a systematic presentation, and synthesis, of the characteristics and findings of the included studies.

6. Process Formulate research question
Determine inclusion and exclusion criteria
Literature search
Pilot test inclusion criteria
Screening – Title/Abstract
Screening – Full-text
Develop and test data abstraction form
Abstract data
Study appraisal
Analysis (meta-analysis)
Synthesis
Prepare Manuscript
Disseminate Findings

7. What is a Rapid Review?
A rapid review is similar to a systematic review but with various constraints
Time
Geography
Sources
Language

8. Challenges for the social Sciences
Systematic reviews are very resource intensive in terms of time and costs
Other issues
Not all the evidence may be in academic database
Not all the evidence is well indexed
Data and methods may not be well described
Meta-analysis may not be possible
Findings are often very broad, not useful for policy makers

9. in the social Sciences
Evaluate for whom the intervention works, when and in what context.
Understanding the political, economic, social and institutional factors are essential to understanding why particular interventions work in particular places at particular times = context matters
Answer the question in a way that makes sense to those who need to know = who is your audience?
Why something works is as important as whether it works

10. What to do? Maintain principles, remain flexible Rigor Transparency
Replicability
Rigor: within the boundaries that you have/that you set
Transparency: should be able to maintain this
Replicability: if you track everything this will follow

11. Process Formulate research question
Determine inclusion and exclusion criteria
Literature search
Screening
Data abstraction
Study appraisal
Analysis (meta-analysis)
Synthesis
What is your object of study? population
What is happening? intervention
What outcome are you trying to evaluate? outcome
2. Criteria: need some background information to get you started
What types of interventions are included?
Which participants should interventions be aimed at?
What kinds of outcome data should be reported?
At this stage, criteria can also be formulated for inclusion and exclusion of study designs and methodological quality.
3. Search for potential studies should be explicitly described.
Databases
Study bibliographies
Websites
Personal communications
Manual searches
4.Screen for inclusion criteria
If done in duplicate need a dispute resolution mechanism
5. Data abstraction/extraction all outcomes of interest
6. Critical appraisal checklists
The skill in critical appraisal lies not in identifying problems, but in identifying errors that are large enough to affect how the result of the study should be interpreted = fit for purpose
Context matters- interpret outcome in terms of context (social, economic, political, institutional factors) to determine transferability
TAPTUPAS
Transparency - are the reasons for it clear?
Accuracy - is it honestly based on relevant evidence?
Purposivity - is the method used suitable for the aims of the work?
Utility - does it provide answers to the questions it set?
Propriety - is it legal and ethical?
Accessibility - can you understand it?
Specificity - does it meet the quality standards already used for this type of knowledge?
7. Analysis – what type of analysis is appropriate?
Group together
8. Interpretation of results
Limitations
Strength of evidence
Applicability
Implications for future research

12. Gathering the Evidence
Aleksandra Blake Trish O’Flaherty Francis Montgomery

13. Tools Literature tracking tool Search Strategy Tracking
PRISMA flow chart
PRISMA
Data abstraction
Note taking
Spreadsheet
Software - eg/ NVIVO
Included/Excluded Studies
List of Studies

14. Group Work Organization Workload ability/flexibility Task assignment
Deadlines
Communicaton
Need to make sure everyone is on the same page: Define, define, define
Document procedures/decisions
Online tools
Basecamp
Dropbox
Google docs
Reference Management software

15. Resources Systematic Reviews in the Social Sciences: A Practical Guide
Campbell Collaboration
International Initiative for Impact Evaluation