An exploration of methods and context for the production of rapid reviews Lisa Hartling, PhD Alberta Research Center for Health Evidence University of Alberta Evidence-based Practice Center CADTH Symposium April 13, 2015

Work Group Evidence-based Practice Center (EPC) Program, U.S. Agency for Healthcare Research and Quality (AHRQ) Jeanne-Marie Guise, Elisabeth Kato, Johanna Anderson, Naomi Aronson, Suzanne Belinson, Elise Berliner, Donna Dryden, Robin Featherstone, Michelle Foisy, Matthew Mitchell, Makalapua Motu’apuaka, Hussein Noorani, Robin Paynter, Karen Robinson, Karen Schoelles, Craig Umscheid, Evelyn Whitlock ( ) Gerald Gartlehner, Aysegul Gozu, Suzanne Hempel, Karen Lee, Annette Totten, Tim Wilt ( )

Conflicts of Interest Nothing to disclose.

Objectives Characterize rapid reviews and similar products Understand the context in which they are produced Understand methodological guidance and strategies used to make products rapid Describe how they differ from systematic reviews Identify empiric evidence of the impact of different methodological approaches

Methods Literature search: Rapid review methods, empiric evidence, methodological guidance Key Informant interviews: Organizations known to produce rapid reviews Characteristics of rapid products Context and uses for rapid products, current practices, structure and evolution of programs and products

Results Literature search: 468 articles found, 116 reviewed in detail, 53 relevant 8 background, 12 reviews, 30 methods, 2 empiric studies Interviews: 17 interviews with 18 Key Informants US, Australia, Canada, UK, Italy Rapid products: 36 examples from 20 organizations

Organizations Public and for-profit Range of experience: several months to 25 years Most had well-established programs for SRs or HTAs Volume of reports varied: 3-5/year to /year Multi-disciplinary staff, familiarity with SRs, extensive experience and training Staff per review: 0 to 9 (1-3 for shorter and 2-4 for longer timeframes) End users varied: most common was health systems Structured nomination process, direct engagement with end user

Rapid Review Methods Most striking observation was degree of variation Approaches to increase efficiency were not consistent Varied across type of product and timeframe

Rapid Review Methods In general longer timeframes resulted in fewer limitations compared with standard SR methods: Most rapid product (within 1 week): limited comprehensiveness, did not conduct full-text review, limited quality control, minimal synthesis and conclusions/recommendations Longer products (over 3 months): Limitations on comprehensiveness Some reliance on existing syntheses Increase in full-text review but often limits on elements extracted Use of dual screening and extraction varied but not common Increase in quality control (e.g., external review), extent of synthesis, conclusions offered

Rapid Review Methods Categorized by extent of synthesis

Contextual Factors Rapid review is not simply “a mini-SR with corners cut” but very essence differs from standard SR

Philosophical Approaches Product FeaturesRapid ReviewSystematic Review Emphasis/priority End user : provide information to help specific user make a decision Product : comprehensive, unbiased, rigorous product, often with multiple end users Relationship with end user Continuous close relationship with specific end user, iterative communication Arms-length relationship with end users, often separate from process Role of existing syntheses High reliance on SRs Often limited use SRs Organizational features/ staffing Maintaining highly trained staff essential More time/possibility to train staff during review Spectrum of products Broad range depending on time available and user needs Consistent, comprehensive product ScopeMore often focused questionsRange from focused to broad questions

Limitations of Rapid Reviews Limited selection and review of literature Risk of missing evidence Limit thought process, consideration of nuances of the evidence Products may be mistaken as SRs Important to explicitly communicate trade-offs Some organizations have disclaimers to avoid misinterpretation of what product offers

Empiric Evidence Limited 2 studies comparing results of rapid reviews with systematic reviews Do not know if and when to be concerned that conclusions of a rapid review may be wrong Concerns raised about safety outcomes in context of rapid products

Conclusions Much variability in products considered to be “rapid reviews” Products with longer production time show consistent increase in comprehensiveness and standard SR methods Classification according to extent of synthesis rather than production time provides more consistent and useful comparison for methods across products Range of methods driven and supported by very different context than standard SRs, in particular nature of the decision and relationship with end user

Future Directions Empiric evidence: Validity of rapid products Specific methods used to increase efficiencies “Durability” of reports: Updating Transparency in reporting End user perspectives: Acceptability of different approaches/products to support decisionmaking needs

Current EPC Working Group End user perspectives Determine what makes AHRQ end users trust and value an evidence synthesis Determine end user experience with and impressions of different rapid products, in terms of: Strengths and limitations When they might use them (different types of decisions) Trade-offs (what are they willing to accept in terms of information and methods) Risks (wrong answer, missing information)

Thank you. /rapid-review-production.pdf