Implementation Survey Results – Systematic Review Questions Next Steps: Implementation Workshop on Standards for Systematic Reviews and Clinical Practice Guidelines Institute of Medicine Tuesday, May 10, 2011 Presenter: Marguerite Koster Committee on Standards for Systematic Reviews

Stakeholder Survey Overview  Purpose: Solicit input from stakeholder groups re barriers to and opportunities for implementing the SR/CPG standards.  with survey link sent to key SR/CPG stakeholders (N=442) –Registered workshop attendees –Members from IOM SR/CPG and “Knowing What Works” (KWW) committees –Workshop attendees and speakers from earlier public SR/CPG/KWW workshops  Follow-up reminder sent one week before survey closing  Response rate = 8.6% (n=38) –Results may not be generalizable to all workshop attendees

Results: Implementing the SR Standards  More than 80% (25/31) of respondents are considering implementing the SR standards  Most cited reasons not to implement standards: –Don’t have the funding –Don’t have the staff resources –Disagree with standards (too cumbersome, no evidence)  All “non-implementers” stated that the new standards will still impact their organization  Timeline for implementing standards:  33% within 6 months  38% within 12 months  29% longer than 12 months  61% of respondents report the standards will make them more likely to collaborate with others on SRs

Results: Standards for Initiating the SR STANDARD% Plan to Implement % Consider standard difficult to implement Establish a team with appropriate expertise and experience 8030 Manage bias and COI in SR team8818 Ensure user and stakeholder input as the review is designed and conducted 5643 Manage bias and COI for individuals providing input in the SR 7222 Formulate the topic for the SR760 Develop a systematic review protocol8419 Submit the protocol for peer review4842 Make the final protocol publicly available, and add amendments to the protocol in a timely manner 5269

Key Comments: Initiating the SR User/Stakeholder Input –Limited resources for user/stakeholder engagement unless they are volunteers – limits variety of perspectives –No mechanism to involve patients or consumers in SR process –Early stakeholder involvement “a death wish by 1,000 paper cuts” Conflict of Interest: –Difficult to identify and address properly; content experts often have conflicts –Rely on self-report of potential COI from participants outside our organization SR Protocols: –Do PICO, but not full SR protocol – takes additional time/resources –Making protocol publicly available for peer review and responding to comments will be a major burden –No resources to engage a third party to manage peer review

Results: Standards for Finding/Assessing Individual Studies STANDARD% Plan to Implement % Consider standard difficult to implement Conduct a comprehensive systematic search for evidence 7616 Take action to address potentially biased reporting of research results 030 Screen and select studies7211 Document the search720 Manage data collection6824 Critically appraise each study7621

Key Comments: Finding/Assessing Individual Studies Searching for evidence –Difficulty finding skilled librarians/information specialists –Will not use limited time on grey literature, unpublished data, or non-English studies; require evidence from peer reviewed journals Addressing biases in studies –Greatest obstacle is selective outcome reporting & publication bias –Without access to prospectively-registered protocols, it is difficult to know whether we have all of the relevant data Screening/selecting studies –Dual reviewing will require additional time and resources; will likely affect the number of SRs we are able to conduct with existing resources Critical appraisal of studies –Critical appraisal of individual studies by each important outcome will require more time and resources

Results: Standards for Synthesizing the Body of Evidence STANDARD% Plan to Implement % Consider standard difficult to implement Use a prespecified method to evaluate the body of evidence 7611 Conduct a qualitative synthesis7616 Decide if a quantitative analysis will be conducted 6418 Conduct a meta-analysis6027 Key Comments: Lack of expertise in conducting meta-analyses

Results: Standards for SR Reporting STANDARD% Plan to Implement % Consider standard difficult to implement Prepare final report with structured format720 Peer review the draft report7211 Publish the final report in a manner that ensures free public access 6033 Key Comments: Business model involves earning money from SRs...if they are free, what's the new model? SR publication is up to the journal – cannot ensure that all SRs will be free and open access for the first 12 months Issues re adherence to confidentiality and journal publication embargoes Implementing public access through a web site or other venue would require additional time and resources

Key Comments: Ways to Overcome Barriers Infrastructure support for the labor-intensive tasks; increase efficiency of existing staff Add more methodologists to the panels; prepare an educational program on conducting SRs and MAs for those and other panelists Conduct fewer SRs and choose topics more judiciously May have to start with dual reviewing for only a percentage of abstracts vs. the entire body of abstracts Will have to be more careful in selection of important outcomes, as each outcome must now be evaluated across the body of evidence Be transparent about any shortcomings in the SR Ensure contractors include these steps in their process Continue to register SRs with Cochrane and Campbell Collaborations Rigorous COI disclosure; ensure balanced input (i.e.,majority of input from those without COI)

Key Comments: Support/Resources Needed A trusted third party to manage the peer review process without charge More help from AHRQ with comparative effectiveness reviews Access to trial protocols Database of vetted stakeholders and experts Training to meet the IOM SR standards, e.g., AHRQ, Cochrane or other organizations that meet the standards could host online/web methodology workshops to assist organizations in meeting standards Being able to provide input to AHRQ on SR topic selection, especially on topics of importance to healthcare organizations/practicing MDs More funding/grants to conduct SRs Wider distribution of standards at the national level to ensure they are recognized and implemented. More specifics on some standards (methods, timing of public access)

Key Comments: Preferences for Collaboration Collaboration in composition of review teams With agreement on IOM SR Standards, adopt other organizations' work and divide the topics for SRs among collaborators Have a national group (PCORI, AHRQ, etc.) actively sponsor, fund and/or organize a collaborative effort, rather than leaving it to individual groups to organize themselves Have an ongoing collaboration, involving multiple stakeholder groups, that will agree on topic selection for SR development, methodology standards (to meet IOM reports), sharing of resources, etc. Work with organizations such as AHRQ and the Cochrane and Campbell Collaborations, to prepare systematic reviews/meta-analyses as a foundation for guidelines, using GRADE methodology A dedicated, funded collaboration site that can provide the infrastructure for collaboration

Thanks to all survey participants!