Standards for Systematic Reviews of Clinical Effectiveness Research Standards for Systematic Reviews of Clinical Effectiveness Research Institute of Medicine January 14, 2010 Institute of Medicine January 14, 2010 Sandra Zelman Lewis, PhD Asst VP, Health & Science Policy American College of Chest Physicians Sandra Zelman Lewis, PhD Asst VP, Health & Science Policy American College of Chest Physicians

Who Produces ACCP SRs? 1.Does your organization produce its own SRs? To what extent does your organization participate in the review? o ACCP uses 3 models: Proposal to AHRQ for SR on narrow subtopic.Membership on TEP.Development of PICO questions.Review of final report ACCP issues RFP to evidence centers for direct contracting.Development of PICO questions.Oversight of process ACCP in-house methodologist.Infrequently.Modeled after AHRQ process 1.Does your organization produce its own SRs? To what extent does your organization participate in the review? o ACCP uses 3 models: Proposal to AHRQ for SR on narrow subtopic.Membership on TEP.Development of PICO questions.Review of final report ACCP issues RFP to evidence centers for direct contracting.Development of PICO questions.Oversight of process ACCP in-house methodologist.Infrequently.Modeled after AHRQ process

Challenges 2.What are the greatest challenges in using SRs to develop CPGs? o Dearth of SRs in areas of clinical interest/need o Appropriate patient populations (including those with comorbidities) not always represented o Quality of process: study limitations, imprecision, inconsistency of results, indirectness of evidence, and publication bias 2.What are the greatest challenges in using SRs to develop CPGs? o Dearth of SRs in areas of clinical interest/need o Appropriate patient populations (including those with comorbidities) not always represented o Quality of process: study limitations, imprecision, inconsistency of results, indirectness of evidence, and publication bias

Quality 4.Do you use any specific instruments or methods to ensure the quality of your SRs? o Assess based on GRADE Evidence Profile criteria:.Study limitations.Imprecision.Inconsistency of results.Indirectness of evidence.Publication bias 4.Do you use any specific instruments or methods to ensure the quality of your SRs? o Assess based on GRADE Evidence Profile criteria:.Study limitations.Imprecision.Inconsistency of results.Indirectness of evidence.Publication bias

Funding 3.How are your SRs funded? Do you accept industry funding? How do you identify and address potential conflict of interest? o Funded through ACCP general funds, some NIH grants, and/or some industry support with solid firewalls:.Supporters have no say in panel chair, composition, evidence center, reviewers, or project staff.No influence on scope or PICO questions.Not on any calls or at any meetings (no knowledge of them at all).No review of any manuscript; first view is publication.Panelists, evidence center staff, etc not informed of sources of support.All panelists, EPC staff, reviewers, etc must pass rigorous COI review (financial and intellectual) 3.How are your SRs funded? Do you accept industry funding? How do you identify and address potential conflict of interest? o Funded through ACCP general funds, some NIH grants, and/or some industry support with solid firewalls:.Supporters have no say in panel chair, composition, evidence center, reviewers, or project staff.No influence on scope or PICO questions.Not on any calls or at any meetings (no knowledge of them at all).No review of any manuscript; first view is publication.Panelists, evidence center staff, etc not informed of sources of support.All panelists, EPC staff, reviewers, etc must pass rigorous COI review (financial and intellectual)

ACCP Conflict of Interest Process 1. Initial Disclosures and Vetting: During the guideline panel member nomination phase 1. Initial Disclosures and Vetting: During the guideline panel member nomination phase 2. Subsequent Disclosures and Vetting: Prior to all face-to-face meetings and on regular schedule 2. Subsequent Disclosures and Vetting: Prior to all face-to-face meetings and on regular schedule 3. Reviewers’ Disclosures and Vetting: Disclosures of all individuals appointed as reviewers 3. Reviewers’ Disclosures and Vetting: Disclosures of all individuals appointed as reviewers 4. Final Disclosures: Updated at time of submission for publication 4. Final Disclosures: Updated at time of submission for publication

ACCP Conflict of Interest Evaluation Process YES Individual has a conflict of interest to disclose: Review by Policy and Procedures Subcommittee, Chair, Vice-Chair, and/ or other designee and presented to full HSP Committee Individual has a conflict of interest to disclose: Review by Policy and Procedures Subcommittee, Chair, Vice-Chair, and/ or other designee and presented to full HSP Committee Recommendation 3: Unacceptable Member is prohibited from participation Recommendation 3: Unacceptable Member is prohibited from participation Recommendation 1: Participation Permitted COI not a source of bias Recommendation 1: Participation Permitted COI not a source of bias Recommendation 2: Participation permitted With precepts to preclude bias Recommendation 2: Participation permitted With precepts to preclude bias Participation permitted Individual has nothing to disclose, evaluation completed Participation permitted Individual has nothing to disclose, evaluation completed Individual Prohibited from Participation Conflict of Interest Disclosure Form Completed No Recommendation 4: Referral to ACCP Executive COI Committee Recommendation 4: Referral to ACCP Executive COI Committee

Setting Standards 5.This IOM committee is charged with recommending standards for SRs of clinical effectiveness research. Are there steps in your SR process that could be standardized? o Use of evidence profiles to focus on the clinical outcomes o Assessment of standard quality measures (as listed above) 6.What would be the implications for your organization if IOM were to recommend a standard grading scheme for characterizing the strength of evidence? o We would welcome the opportunity to review and consider applying for future guidelines not yet in process. o We would also welcome the opportunity to participate in future development of this grading scheme. 5.This IOM committee is charged with recommending standards for SRs of clinical effectiveness research. Are there steps in your SR process that could be standardized? o Use of evidence profiles to focus on the clinical outcomes o Assessment of standard quality measures (as listed above) 6.What would be the implications for your organization if IOM were to recommend a standard grading scheme for characterizing the strength of evidence? o We would welcome the opportunity to review and consider applying for future guidelines not yet in process. o We would also welcome the opportunity to participate in future development of this grading scheme.

The Future The IOM report and new standards should be presented to appropriate audiences: HTA and SR producers, evidence center methodologists, guideline developers and implementers will be attending the Guidelines International Network conference (see next slide) and guideline methodology course

Guidelines International Network (G-I-N) 2010 Conference - Chicago August 26-28, 2010 – Conference Dates August 25, 2010 – Pre-meeting Course in Guideline Methodology Chicago, Illinois, USA Host: American College of Chest Physicians