1

Objectives Describe ways in which the EPC program has engaged stakeholders to improve relevance and applicability of its products Present methods used for stakeholder engagement Identify challenges in stakeholder engagement and strategies employed by the EPC program

Evidence-based Practice Centers (EPCs)  11 centers throughout the United States and Canada Systematically and critically appraise existing research and synthesize knowledge Produce comparative effectiveness and effectiveness reviews Scientific Resource Center (SRC) Responsible for scientific methodological work of reviews and other research projects for EPCs and DEcIDES EPCs & SRC

Blue Cross and Blue Shield Association, Technology Evaluation Center, Chicago, IL Brown University, Providence Rhode Island ECRI Institute, Plymouth Meeting, PA Johns Hopkins University, Baltimore, MD Kaiser Foundation Research Institute, Oakland, CA Oregon Health & Science University, Portland, OR RTI International – University of North Carolina at Chapel Hill, Chapel Hill, NC RAND Corporation, Santa Monica, CA University of Alberta, Edmonton, Edmonton, Alberta University of Minnesota EPC, Minneapolis, MN Vanderbilt University Medical Center, Nashville, TN Scientific Resource Center (SRC) for EPCs and DEcIDES – Portland VA Research Foundation, Portland OR

Public Nominate Review Topics Technical Expert Panel Key Research Questions Peer and Public Review Feedback on Summary Guides Future Research Disseminate Reports Key Informants Opportunities for Stakeholder Input

EHC Report on Engaging Stakeholders for Development and Prioritization of Future Research Needs O’Haire C, McPheeters M, Nakamoto E, LaBrant L, Most C, Lee K, Graham E, Cottrell E, Guise J-M. Engaging Stakeholders To Identify and Prioritize Future Research Needs. Rockville (MD): Agency for Healthcare Research and Quality (US); 2011 Jun. (Methods Future Research Needs Reports, No. 4.) Available from:

Literature Synthesis Phase III: EPC interviews (Vanderbilt EPC) Recruitment and Interview Guide Development Analysis of Interviews Key Informant Interviews Phase II: Key Informant Interviews (Oregon EPC) Phase I: Literature Summary (Oregon EPC) Literature Review Literature Search Cumulative Synthesis EPC Protocol Review Invitations and EPC Discussion Guide EPC Discussions Analysis of Discussions

Stakeholder Identification & Recruitment Identify pertinent stakeholder groups (intentional based upon stakeholder groupings) Ensure a balance of stakeholder perspectives (i.e. consumer, clinician, researcher, research funder, insurer/payer, manufacturer) The results of EPC FRN pilot projects suggest that engaging 6-10 stakeholders is appropriate OMB restricts the number of participants in surveys to no more than 9 non-federal employees Research Needs Development

Stakeholders Stakeholder Group Description Consumer (public, patient, and caregiver) An individual or advocacy group representing individuals who have a health condition, use health care services and/or who are members of the community Clinician (Health professional) Health care or public health provider (e.g., academic, rural/frontier, and community) and medical and/or public health organizations Social Service Organizations An individual or organization that advances human welfare or social work (e.g. community or condition service providers, school-based programs, justice system) Policymaker An individual or organization who is involved in health care policy (e.g. local, state, provincial and Federal legislators and staff)  Medical organizations  Governmental organizations (e.g. VA, AHRQ, etc.) Researcher An individual who conducts and/or facilitates research activities in:  Basic, translational, clinical sciences; research methodology, public health or health services, and systematic reviews Research Funder A public or private organization that funds research (e.g. National Institute of Health, Susan G. Komen Foundation, and American Cancer Society) Insurer/Payer An organization or agency that pays for health-related goods and services (e.g. Blue Cross Blue Shield, Medicaid, and Medicare) or a business group that pays for health insurance (e.g. employers and government) Manufacturer A business group that produces health-related items (e.g. pharmaceuticals and medical devices)

Methods of Stakeholder Engagement Engagement  Most common: Focus Groups, Symposia/Conferences Prioritization  Most common: Priority Questionnaire, Consensus Engagement & Prioritization Simultaneously  Priority Survey  Delphi

+ GOOD – FAIR POOR

Topic Generation Nominate topics via the web Topic Development Clarify nomination intent Frame clinical context of nomination Topic Refinement Provide “real world” input Provide input on clinical logic Public comment via Web Review Provide methodological & clinical input Provide input via peer review Provide public comment via Web Research Needs Development Participate in discussions to describe research gaps & prioritize research needs Report Translation & Dissemination Participate in focus groups Participate in individual interviews Provide expert review of summary guides EHC Program Research Reviews Topics Evidence Review ResearchD&I

Topic Generation Approaches Nominal Group Process  In-person  Idea generating  Intended to promote group participation Our experience: In-person brainstorming with flip chart used to document topics Each state had 8 stickers to vote for priority topics: #1 (2 red dots) #2 (2 green dots) #3 (2 blue dots), and #4 (2 yellow dots) Weighting system applied to priorities: #1 - 4 points for each vote #2 - 3 points #3 – 2 points #4 – 1 point

Topic Generation Approaches Medicaid Medical Directors Analytic Process: Weighting system : #1 - 4 points #2 - 3 points #3 – 2 points #4 – 1 point

Prevention in Women’s Health Modified Process: Local Key Informants Three web meetings: May 11 – Framework & Initial Topics (Extranet & topic generation) June 8 – Topic Generation (Evidence Scan)  13 – Final Prioritization Prioritization Survey – survey monkey Web whiteboard during 3 rd meeting Topic Generation Approaches

Stakeholders Engaged in:  Development of CER key questions  Refinement of topic  Review of Draft Methods used:  Group or individual calls  Web-based forums with survey type format to provide responses Scoping CER

Preparation of Stakeholders is critical: Assume that all stakeholders need orientation to the CER process Very important for them to know where they fit in and what to expect from their feedback Familiarity with research methods likely to be very variable Conflicts of interest are to be expected with stakeholder groups, but must be disclosed. Scoping CER

Topic Generation Nominate topics via the web Topic Development Clarify nomination intent Frame clinical context of nomination Topic Refinement Provide “real world” input from provider, patient, and policy perspectives Provide input on clinical logic Provide public comment via Web Research Review Provide methodological & clinical input as a TEP member Provide input via peer review Provide public comment via Web Research Needs Development Participate in discussions to describe research gaps & prioritize research needs Report Translation & Dissemination Participate in focus groups Participate in individual interviews Provide expert review of summary guides EHC Program Research Reviews Stakeholder Preparatory Materials

Where We Are in the Comparative Effectiveness Review Process Name of PanelEPC Role & What to Expect Describe what has occurred to date Describe Intent of panel Numbers and types of individuals Numbers of times and ways they may be engaged List ways TEP is especially helpful to process The role of the Evidence-based Practice Center (EPC) is to ……. List Key Principles Reviews commissioned under the Effective Healthcare Program (EHC) are posted publically at different stages of the review process, including at the stage of proposed Key Questions and the draft report stage. Stakeholder Preparatory Materials

Comparative Effectiveness Reviews (CERs) critically appraise and synthesize bodies of literature and identify areas where research is lacking or insufficient in either quantity or quality There are limitations in funds available to support research To fully actualize their potential, CERs should be utilized to inform and guide future research Future Research Needs (FRN) documents are designed to facilitate this process by engaging stakeholders to prioritize future research topics Future Research Needs Development Purpose

Future Research Needs Development Step 1 Research gaps from CER interviews Step 2 Identify & Recruit Stakeholders Step 3 Orient stakeholders to project Step 4 Solicit input on gaps, generate topics Step 5 Engage stakeholders to prioritize FRN (at least 2 rounds) Step 6 Categorize by PICOs Step 7 Search for ongoing Research & Reviews Step 8 Write up report Step 9 Elicit stakeholder feedback Step 10 Final report

Combined Methods: Semi-structured interviews to identify topics Modified Delphi process for prioritization Semi-structured interviews  Telephone  Conference calls  Webinars Delphi #1 Survey Topics organized by PICOs Likert Scale – 6-point scale of no, low, highest priority #2 Survey Likert Ranking Indicate top 3 in order Research Needs Development

Priority Final Rank Weighted Score a Studies to test clinical, institutional, or policy interventions to increase access to “safe” TOL 172 Research on barriers to providing safe TOL, including factors that limit the ability of hospitals to meet the “immediately available” requirement 251 Studies comparing outcomes for mother and infant in settings where physicians are “immediately available” vs. settings where physicians are “readily available” 346 Studies to understand best practice models based on institutions that are currently offering safe TOL 441 Development of standardized measures for short-and long-term maternal and infant outcomes 541 Surveillance to determine long-term clinical outcomes of TOL vs. ERCD 638 Research on how patients understand risk, how they respond to different ways of framing risk, and how best to communicate risks of TOL vs. ERCD 737 Clinical and policy relevant studies to address the threat of legal liability on practice patterns regarding TOL vs. ERCD 834 Development/utilization of a reliable model or tool to predict the probability of successful VBAC for individual women /tool to predict probability of successful VBAC 932 Studies to refine, validate, and implement informed consent templates that are informative, reliable, and able to be well documented 1026

RESEARCH CONTINUUM Topic Generation Study Design Implementation Analysis/ Interpretation Research Prioritization Dissemination Clinicians Consumers Funders Insurers Researchers Policymakers Clinicians Consumers Funders Insurers Manufacturers Researchers Payers Policymakers Clinicians Manufacturers Researchers Policymakers Clinicians Consumers Funders Insurers Manufacturers Researchers Payers Policymakers Clinicians Consumers Researchers Policymakers Clinicians Researchers EPC Stakeholder Engagement Expanded Scope of Future Research Needs Beyond CER Research Needs Development

Summary  Several options for methods to use when engaging stakeholders  Method depends on several factors including Intent (identifying topics, prioritization) Resources Number and variety of stakeholders What is already known in topic Experience  Important to outline approach and intent a priori  Relationships with stakeholders are important Contact potential stakeholders through multiple venues Consistent communication and follow-up in the form of s or phone calls is important to ensuring a balanced selection of stakeholders and a high response rate  Preparation of stakeholders is critical to define expectations and allow engagement