Systematic and Rapid ReviewS

Where did they come from? Health sector Research evidence not being included in doctor’s decision-making Preventable deaths occurred Spread to other disciplines: education, environmental sciences, engineering, international development (Antman et al. 1992) The paper showed that traditional review articles and textbooks had often given treatment advice that was dangerously inconsistent with the evidence available at the time they had been written. “Review articles often failed to mention important advances or exhibited delays in recommending effective preventive measures. In some cases, treatments that have no effect on mortality or are potentially harmful continued to be recommended by several clinical experts.” http://www.jameslindlibrary.org/antman-em-lau-j-kupelnick-b-mosteller-f-chalmers-tc-1992/ (IC) was summoned to give evidence to a House of Lords Committee on medical research, and drew on the paper’s findings. The Committee was informed that, five years after a systematic review of controlled trials had shown that thrombolysis reduced the risk of death after myocardial infarction, the Oxford Textbook of Medicine maintained that the benefits of the treatment had not been established (Pentecost 1987) . The following weekend (5 Feb 1995), this contribution to the Committee’s thinking led The Sunday Times to publish an article on its front page under the headline ‘Hundreds killed by doctors relying on outdated manuals

Policy Horizons Canada Why bother? “Evidence-based, evidence-informed or knowledge-based policy development refers to an approach that levers the best available objective evidence from research to identify and understand issues so that policies can be crafted by decision makers that will deliver desired outcomes effectively, with a minimal margin of error and reduced risk of unintended consequences.” Policy Horizons Canada

What are they? Type of literature review Asks a specific question Rigorous Transparent Replicable attempt to summarize all pertinent studies on a specific question can improve the understanding of inconsistencies in diverse evidence can identify gaps in research evidence to define future research agendas Question usually about the effectiveness of a program or treatment Rigorous – very thorough, try to gather all relevant evidence, assess the quality of research Transparent – stipulate apriori the protocol (state assumptions), recognize methodological biases Replicable – results can be reproduced

What is a systematic review? A literature review that asks a specific question and has a clearly stated set of objectives with pre-defined eligibility criteria for studies; an explicit, reproducible methodology; a systematic search that attempts to identify all studies that would meet the eligibility criteria; an assessment of the validity of the findings of the included studies; a systematic presentation, and synthesis, of the characteristics and findings of the included studies.

Process Formulate research question Determine inclusion and exclusion criteria Literature search Pilot test inclusion criteria Screening – Title/Abstract Screening – Full-text Develop and test data abstraction form Abstract data Study appraisal Analysis (meta-analysis) Synthesis Prepare Manuscript Disseminate Findings

What is a Rapid Review? A rapid review is similar to a systematic review but with various constraints Time Geography Sources Language

Challenges for the social Sciences Systematic reviews are very resource intensive in terms of time and costs Other issues Not all the evidence may be in academic database Not all the evidence is well indexed Data and methods may not be well described Meta-analysis may not be possible Findings are often very broad, not useful for policy makers

in the social Sciences Evaluate for whom the intervention works, when and in what context. Understanding the political, economic, social and institutional factors are essential to understanding why particular interventions work in particular places at particular times = context matters Answer the question in a way that makes sense to those who need to know = who is your audience? Why something works is as important as whether it works

What to do? Maintain principles, remain flexible Rigor Transparency Replicability Rigor: within the boundaries that you have/that you set Transparency: should be able to maintain this Replicability: if you track everything this will follow

Process Formulate research question Determine inclusion and exclusion criteria Literature search Screening Data abstraction Study appraisal Analysis (meta-analysis) Synthesis What is your object of study? population What is happening? intervention What outcome are you trying to evaluate? outcome 2. Criteria: need some background information to get you started What types of interventions are included? Which participants should interventions be aimed at? What kinds of outcome data should be reported? At this stage, criteria can also be formulated for inclusion and exclusion of study designs and methodological quality. 3. Search for potential studies should be explicitly described. Databases Study bibliographies Websites Personal communications Manual searches 4.Screen for inclusion criteria If done in duplicate need a dispute resolution mechanism 5. Data abstraction/extraction all outcomes of interest 6. Critical appraisal checklists The skill in critical appraisal lies not in identifying problems, but in identifying errors that are large enough to affect how the result of the study should be interpreted = fit for purpose Context matters- interpret outcome in terms of context (social, economic, political, institutional factors) to determine transferability TAPTUPAS Transparency - are the reasons for it clear? Accuracy - is it honestly based on relevant evidence? Purposivity - is the method used suitable for the aims of the work? Utility - does it provide answers to the questions it set? Propriety - is it legal and ethical? Accessibility - can you understand it? Specificity - does it meet the quality standards already used for this type of knowledge? http://www.scie.org.uk/publications/knowledgereviews/kr03.asp 7. Analysis – what type of analysis is appropriate? Group together 8. Interpretation of results Limitations Strength of evidence Applicability Implications for future research

Gathering the Evidence Aleksandra Blake Trish O’Flaherty Francis Montgomery

Tools Literature tracking tool Search Strategy Tracking PRISMA flow chart PRISMA Data abstraction Note taking Spreadsheet Software - eg/ NVIVO Included/Excluded Studies List of Studies

Group Work Organization Workload ability/flexibility Task assignment Deadlines Communicaton Need to make sure everyone is on the same page: Define, define, define Document procedures/decisions Online tools Basecamp Dropbox Google docs Reference Management software

Resources Systematic Reviews in the Social Sciences: A Practical Guide Campbell Collaboration International Initiative for Impact Evaluation