1. Engaging Stakeholders in the Effective Health Care Program Information and tools for researchers and investigators

2. Full Course Outline Module II: Stakeholders and Stakeholder Engagement Module III: Points of Engagement in the EHC Program Module IV: Skills for Successful Engagement

3. Module I AHRQ’s Vision for Stakeholder Engagement

4. What to Expect in Module I Module I will cover the following topics:  AHRQ’s vision for stakeholder engagement in the EHC Program  The theory and evidence behind stakeholder engagement in comparative effectiveness research  The alignment of principles of comparative effectiveness research (CER) and stakeholder engagement

5. Module I Learning Objectives After completing Module I, participants will be able to:  Identify AHRQ’s vision for stakeholder engagement  Describe the theoretical basis for engaging stakeholders  List at least three benefits of engaging stakeholders in research  Identify three differences between knowledge transfer and knowledge exchange

6. Stakeholder Engagement in Research

7. Defining Stakeholder Engagement

8. AHRQ’s Vision for Stakeholder Engagement

9.  Stakeholders are involved throughout the research process  Create strategic involvement and partnerships with a broad and inclusive range of stakeholders  Collaborative relationships should inform all phases of research  Stakeholder input maximize usefulness of EHC Program products

10. Theory of Stakeholder Engagement Evidence suggests that engaging stakeholders in research increases:

11. Theory of Stakeholder Engagement: Relevance & Context Studies indicate that involving stakeholders early on leads to research products that feature:  Relevant information  Applicable evidence  Real-world context  Information to address existing stakeholder concerns What are the best oral medicines to treat for type 2 diabetes? - Topic submitted by the American College of Physicians, a professional association representing 132,000 medical practitioners across the nation

12. Theory of Stakeholder Engagement: Knowledge Exchange Involving stakeholders in the research process facilitates:

13. Knowledge Transfer vs. Knowledge Exchange One-way Communicates intent and/or results Single or limited opportunities Before or after Linear Prescribed Individual benefit/expectation Two-way Engages in dialogue, questions & exploration Multiple opportunities Before, during & after Iterative Organic Mutual benefit/expectation

14. Engagement Exercise 1: Stakeholder Engagement Techniques Think about which of the following stakeholder involvement techniques have been used by you or your research team:  Discussing/identifying potential research topics with stakeholders  Attending meetings of stakeholder groups/organizations  Presenting information or findings at meetings of stakeholder organizations  Asking clinical experts to help develop or clarify a research suggestion  Involving stakeholders in developing research questions  Involving patients & consumers or frontline clinicians in topic refinement activities  Posting draft documents for review and comment  Publishing findings in journals, newsletter, etc.

15. Engagement Exercise 2: Knowledge Transfer vs. Knowledge Exchange The following slide provides examples of stakeholder engagement techniques. For each example, please consider if the activity is a form of knowledge transfer or knowledge exchange.

16. Engagement Exercise 2: Knowledge Transfer vs. Knowledge Exchange  Discussing/soliciting potential research questions with stakeholders  Attending meetings of stakeholder groups/organizations  Presenting information or findings at meetings of stakeholder groups/organizations  Involving a nominator in topic refinement activities  Posting a draft document for review & comment  Publishing findings in journals, newsletters, etc.

17. Engagement Exercise 2: Answers Knowledge Transfer vs. Knowledge Exchange Knowledge Exchange  Discussing/soliciting potential research questions with stakeholders  Involving a nominator in topic refinement activities  Posting a draft document for review & comment Knowledge Transfer  Attending meetings of stakeholder groups/organizations  Presenting information or findings at meetings of stakeholder groups/organizations  Publishing findings in journals, newsletters, etc.

18. Theory of Stakeholder Engagement: Application of Evidence Involving stakeholders in the research process leads to applicable research products that: Address specific questions with appropriate context Are useable and actionable Incorporate wider contextual issues Are accepted and valued by end-users

19. Theory of Stakeholder Engagement: Dissemination Avenues Involving stakeholders in the research process can lead to dissemination opportunities through:  Advocacy organizations  Professional societies  Family and friends  Clinicians  Health care systems  Payers  Public programs

20. Engagement Exercise 3: The following slide provide examples of engagement activities. Please consider the possible resulting benefit(s) for each example.

21. Engagement Exercise 3: Please consider the possible resulting benefit(s) of stakeholder engagement: Discussing/soliciting potential research ideas from stakeholders. a)Increased relevance & context b)Knowledge exchange c)Increased application of evidence d)Increased dissemination avenues

22. Engagement Exercise 3: Benefits of Stakeholder Engagement Answer: A, B, C & D Discussing/soliciting potential research ideas from stakeholders. a)Increased relevance & context b)Knowledge exchange c)Increased application of evidence d)Increased dissemination avenues

23. Engagement Exercise 3: Please consider the possible resulting benefit(s) of stakeholder engagement: Attending meetings of stakeholder groups/ organizations a)Increased relevance & context b)Knowledge exchange c)Increased application of evidence d)Increased dissemination avenues

24. Engagement Exercise 3: Answer: A & D Attending meetings of stakeholder groups/ organizations : a)Increased relevance & context b)Knowledge exchange c)Increased application of evidence d)Increased dissemination avenues

25. Engagement Exercise 3: Please consider the possible resulting benefit(s) of stakeholder engagement: Presenting information of research findings at meetings of stakeholder groups/organizations a)Increased relevance & context b)Knowledge exchange c)Increased application of evidence d)Increased dissemination avenues

26. Engagement Exercise 3: Answer: C & D Presenting information of research findings at meetings of stakeholder groups/organizations a)Increased relevance & context b)Knowledge exchange c)Increased application of evidence d)Increased dissemination avenues

27. Engagement Exercise 3: Please consider the possible resulting benefit(s) of stakeholder engagement: Involving stakeholders in topic refinement activities a)Increased relevance & context b)Knowledge exchange c)Increased application of evidence d)Increased dissemination avenues

28. Engagement Exercise 3: Answer: A, B, C & D Involving stakeholders in topic refinement activities a)Increased relevance & context b)Knowledge exchange c)Increased application of evidence d)Increased dissemination avenues

29. Engagement Exercise 3: Please consider the possible resulting benefit(s): Posting draft documents for review & comments a)Increased relevance & context b)Knowledge exchange c)Increased application of evidence d)Increased dissemination avenues

30. Engagement Exercise 3: Answer: A, B, C & D Posting draft documents for review & comments a)Increased relevance & context b)Knowledge exchange c)Increased application of evidence d)Increased dissemination avenues

31. EHC Program Principles Key principles identified by the EHC Program to guide research activities include:  Research must be relevant and timely to meet the needs of decision-makers  Public participation and transparency increase public confidence in the scientific integrity, credibility of research processes, and provide further accountability to researchers (Slutsky, et al., 2008)

32. Patient-Centered Approach to CER  Central to evidence-based medicine  Research should address patient & consumer concerns  Requires different techniques

33. AHRQ & Patient-Centered Approach to CER One of the overarching principles of the EHC Program is to conduct CERs by approaching the evidence by a clinical, patient-centered perspective. (Helfand & Balshem,2009; Slutsky, et. al., 2008)

34. Engagement Exercise 4: Patient-Centered Research There are many ways researchers can demonstrate that their work is patient-centered. Please think of three examples of how your work has been patient-centered.

35. Examples of Patient-Centered Approaches  Providing public access and opportunity to comment on draft documents  Focusing reports on outcomes and measures important to patients and their clinicians  Documenting and making publicly available the processes used to develop a report

36. Examples of Patient-Centered Approaches (Cont)  Involving patients and their clinicians in:  Generating and prioritizing research topics and questions  Identifying and refining key questions  Planning, developing and updating reviews and protocols/reports  Consulting during report development  Contributing to other dissemination & translational activities  Collecting patient “testimonials” that illustrate the context and circumstances of patients

37. Summary of Module I  AHRQ’s vision for stakeholder engagement is inclusive, broad, and evolving  AHRQ’s expectation for stakeholder engagement is outlined in the initial legislation and current ARRA spend plan  Evidence suggests there are many benefits to engaging stakeholders in research  Principles of comparative effectiveness research (CER) and stakeholder engagement align closely

38. Module I Quiz To help you review the information presented in Module I, please complete the following quiz. Answers are provided at the end of the module.

39. Please select the statement that most closely represents AHRQ's vision for stakeholder engagement in the EHC Program: A.Stakeholders should only be involved in the final stages of research B.Stakeholders should be strategically involved throughout the research process C.Stakeholder should be silent partners D.Stakeholders impede the research process, but have to be included according to Congress Module I Quiz: Question 1 Module I Quiz: Question 1

40. Please select the statement that most closely represents AHRQ's vision for stakeholder engagement in the EHC Program: A.Stakeholders should only be involved in the final stages of research B.Stakeholders should be strategically involved throughout the research process C.Stakeholder should be silent partners D.Stakeholders impede the research process, but have to be included according to Congress Module I Quiz: Question 1: ANSWER Module I Quiz: Question 1: ANSWER

41. Which of the following does the evidence suggest are potential benefits of stakeholder involvement in research? A.Increased application of evidence B.Lower reading level of research reports C.Increased relevancy of research products D.Increased dissemination avenues E.Increased number of editors to review reports Module I Quiz: Question 2 Module I Quiz: Question 2

42. Which of the following does the evidence suggest are potential benefits of stakeholder involvement in research? A.Increased application of evidence B.Lower reading level of research reports C.Increased relevancy of research products D.Increased dissemination avenues E.Increased number of editors to review reports Module I Quiz: Question 2: ANSWER Module I Quiz: Question 2: ANSWER

43. One of the core principles of comparative effectiveness research and evidence-based medicine that closely aligns with principles of stakeholder engagement is: A.Standardize methodology B.Peer review process C.Patient-centered approach Module I Quiz: Question 3 Module I Quiz: Question 3

44. One of the core principles of comparative effectiveness research and evidence-based medicine that closely aligns with principles of stakeholder engagement is: A.Standardize methodology B.Peer review process C.Patient-centered approach Module I Quiz: Question 3: ANSWER Module I Quiz: Question 3: ANSWER

45. Works Cited Agency for Healthcare Research and Quality (2009). Methods Guide for Effectiveness and Comparative Effectiveness Reviews. Rockville, MD. Available at: http://effectivehealthcare.ahrq.gov/index.cfm/search-for- guides-reviews-and-reports/?pageaction=displayproduct&productid=318. Agency for Healthcare Research and Quality (2009). Operating Plan for American Recovery and Reinvestment Act Funds for Comparative Effectiveness Research as the Agency for Healthcare Research and Quality. Burger, J., Gochfeld, M. & Pletnikoff, K. (2009). Collaboration versus communication: The Department of Energy’s Amchitka Island and the Aleut Community. Environmental Research, 109: 503-510. Canadian Health Services Research Foundation (no date). Knowledge Exchange. Retrieved from http://www.chsrf.ca/knowledge_transfer/index_e.php. Gooberman-Hill, R. & Hoorwood, J. (2008). Citizens’ juries in planning research priorities: process engagement and outcome. Health Expectations, 11:272- 281.

46. Works Cited Graham, I.D., Logan, J., Harrison, M.B., Straus, S.E., Tetroe, J., Caswell, W. & Robinson, N. (2006). Lost in Knowledge Translation: Time for a Map? The Journal of Continuing Education in the Health Professions, 26: 13-24. Guyatt, G. & Rennie, D. (Eds.). (2002). Users’ guides to the medical literature: A manual for evidence-based clinical practice. Chicago, IL: AMA Press. Helfand, M. & Balshem, H. (2009). Principles for developing guidance: AHRQ and the effective health care program. Journal of Clinical Epidemiology. Article in press. Innvaer, S., Vist, G., Trommald, M. & Oxman, A. (2002). Health policy-makers’ perceptions of their use of evidence: a systematic review. Journal of Health Services Research & Policy, 7(4): 239-44. Keown, K., Van Eerd, D. & Irvin, E. (2008). Stakeholder Engagement Opportunities in Systematic Reviews: Knowledge Transfer for Policy and Practice. Journal of Continuing Education in the Health Professions, 28(2): 67-72. Lomas, J. (2000). Using ‘Linkage and Exchange’ to Move Research into Policy at a Canadian Foundation. Health Affairs, 19(3): 236-240.

47. Works Cited Lomas, J. (2000). Using ‘Linkage and Exchange’ to Move Research into Policy at a Canadian Foundation. Health Affairs, 19(3): 236-240. O’Haire C, McPheeters M, Nakamoto EK, LaBrant L, et al. Methods for engaging stakeholders to identify and prioritize future research needs. Methods Future Research Needs Report No. 4. (Prepared by Oregon EPC under Contract No. HHSA 290-2007-10057-I.) AHRQ Publication No. 11-EHC044-EF. Rockville, MD: AHRQ. June 2011. Available at: www.effectivehealthcare. Ahrq.gov/reports/final.cfm. Saunders, C., Crossing, S., Girgis, A., Butow, P. & Penman, A. (2007). Operationalising a model framework for consumer and community participation in health and medical research. Australia and New Zealand Health Policy, 4(13). Slutsky, J., Atkins, D., Chang, S. & Collins Sharp, B.A. (2008). Comparing medical interventions: AHRQ and the effective health-care program. Journal of Clinical Epidemiology. 63: 481-483. Whitlock, E.P., Lopez, S.A., Chang, S., Helfand, M., Eder, M. & Floyd, N. (2010). Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health Care program. Journal of Clinical Epidemiology. 63(5): 491-501.