Approaches for Supporting Evidence- and Values-Informed Policymaking Knowledge, Sharing, Doing: 1st National KT Conference in Rehabilitation 4 May 2016 Montréal, Québec Michael G. Wilson, PhD Assistant Director, McMaster Health Forum Assistant Professor, McMaster University

Evidence- and Values-Informed Policy (1) Systematically and transparently using the best available data and research evidence, as well as citizens’ values and preferences, in each of: Prioritizing problems and understanding their causes (agenda setting) Deciding which option to pursue (policy development) Ensuring the chosen option makes an optimal impact at acceptable cost (policy implementation) … alongside the institutional constraints, interest-group pressure, values and other types of information (like jurisdictional reviews, consultations, expert review groups, and opinion polls) that influence the policy process 2

Evidence- and Values-Informed Policy (2) Source: Hoffman SJ, Rottingen J-A, Bennett S, Lavis JN, Edge JS, Frenk J. Building Health Systems Research as a Field of Scientific Endeavour: Wading through Definitional Confusion, Conceptual Challenges and Opportunities for the Future, Manuscript under view. 3

Options Available to Support the Use of Research Evidence EXTRA 6/3/2018 Options Available to Support the Use of Research Evidence Two factors that consistently increased the prospects for research use in management / policy (based on a systematic review of 124 studies) Citation: Lavis JN, Catallo C, editors (2013). Bridging the worlds of research and policy in European health systems. Copenhagen, WHO Regional Office for Europe. Interactions between researchers and decision-makers Timing/ timeliness of research evidence 4 (c) CHSRF/FCRSS

Examples of Approaches Timeliness Timeliness & interactions Interactions One-stop-shops Rapid-response units Citizen briefs & panels Evidence briefs & stakeholder dialogues Training/capacity building 5

One-Stop-Shops (1) One-stop shops (and evidence services) are a promising ‘self-serve’ KT innovation for several reasons Supports timely access (everything in one place) Facilitates assessments of relevance Organized by priority topics (e.g., health systems) Provide decision-relevant information (e.g., quality, countries in which included studies are conducted) Enhances communication Presentation of evidence in several formats and in ways that are user friendly (e.g. links to free full text or user-friendly summaries) 6

‘My health’ ‘Our health’ ‘Our system’ One-Stop-Shops (2) McMaster PLUS (http://hiru.mcmaster.ca/) Pre-appraised studies and reviews that address the full range of questions about clinical programs and services and about drugs ‘My health’ Health Evidence (www.healthevidence.org) Pre-appraised reviews that address questions about the effectiveness of public health programs ‘Our health’ Health Systems Evidence (www.healthsystemsevidence.org) pre-appraised reviews and economic evaluations that address how to strengthen health systems and get cost-effective programs, services and drugs to those who need them ‘Our system’ 7

One-Stop-Shops (3) 8

Evaluation & Impact One-stop-shops (Health Systems Evidence) 11,596 registered users (4403 signed up to receive a monthly customized evidence service) Researchers = 3874; Policymakers = 2460; Healthcare professionals = 2630; Managers = 1261; Plus more than 3000 students and 2464 ‘other’ Endorsed by WHO’s Health Systems Research Synthesis Group as the one-stop shop for research syntheses about health systems Incorporated into other resources (e.g., EVIPNet Virtual Health Library, McMaster Optimal Aging Portal) Increasingly used to inform high-profile scientific articles / studies Cited as the key source in the New England Journal of Medicine (Mills) and in Health Policy (Rockers et al.) 9

Rapid-Response Units (1) EXTRA 6/3/2018 Rapid-Response Units (1) Often need to address pressing health system issues in days or weeks May need support with finding and synthesizing research evidence given competing demands (but not enough time to prepare an evidence brief and convene a dialogue) Rapid-response units fills a gap between ‘Self-serve’ approaches (e.g., one-stop shops) and ‘Full-serve’ approaches (e.g., stakeholder dialogues informed by evidence briefs) 10 (c) CHSRF/FCRSS

Synthesize relevant evidence Rapid-Response Units (2) There are several existing programs in Canada (e.g., CADTH, INESSS, OHTN), but not specifically focused on health-system questions 11 stakeholder dialogue participants agreed that there is a clear need for such a program Key recommendations: build a model decide what can done in what timelines define success and measure it Refine question Conduct searches Review search results Synthesize relevant evidence Requestor 11

Evaluation & Impact Rapid response Evaluation approach is in development Examples of impacts A synthesis about suicide-prevention interventions directly informed the 2014 Toronto Public Health Suicide Prevention Strategy Three syntheses for the College of Physicians and Surgeons of Ontario directly informed College policies unsafe medical practices preventing sexual abuse educational approaches to improve clinical performance 12

Citizen Panels (1) Key challenges Complex health problems different understandings of the problem Uncertainties e.g., about the most effective policy options to address these problems and their implementation considerations (e.g., equity, costs, unintended effects, acceptability and feasibility) Lack of agreement among all stakeholders about how to move forward 13

Citizen Panels (2) The role of citizens’ values and preferences Citizens can help us to develop a shared understanding about a problem (challenge 1) Citizens have experiential knowledge (challenge 2) valid and legitimate evidence that can help to find innovative and local solutions to complex problems Citizens can facilitate or trigger action (challenge 3) offering guidance on how to move forward identifying what options are socially, politically, and ethically sound advocating for policy options 14

Citizen Panels & Stakeholder Dialogue Process Overview 1. Consulting with key stakeholders 2. Preparing a citizen and evidence brief 3. Convening a citizen panel and preparing a summary 4. Convening a stakeholder dialogue and preparing a summary 5. Supporting action 15

1. Consulting with Stakeholders Steering committee Develop criteria to select diverse panel participants Iteratively refine understanding of the problem, possible options to address it and implementation considerations Identify key informants Key informant interviews 15-20 interviews with representatives of citizen/patient groups, policymakers, stakeholders and researchers who are involved in or affected by the issue 16

2. Preparing a Citizen & Evidence Brief Presents (in plain language) relevant research evidence about a problem, options for addressing it, and key implementation considerations Based on syntheses and local evidence (identified using systematic and transparent approach) No recommendations Subjected to merit review Citizen brief poses questions for citizens to consider 17

3a. Convening a Citizen Panel Brings together a diverse group of 10 to 16 citizens for a one-day, off-the-record, dialogue that provides them with the opportunity to: bring their own views and experiences to bear on an issue; learn from the evidence and from others’ views and experiences; and share their newly informed views about the issue and how to address it. 18

3b. Preparing a Summary Thematic analysis of deliberations Describes: views about and experiences related to the issue values and preferences for addressing it Identify areas of shared understandings, as well as divergent opinions 19

4a. Convening a Stakeholder Dialogue Brings together 18-22 policymakers, stakeholders and researchers for ‘off-the-record’ deliberations about: the problem options to address it implementation considerations next steps Participants chosen because of their ability to: bring unique views and experiences champion actions to address the challenge creatively 20

4b. Preparing a Summary Thematic analysis of deliberations Identifies areas of common ground Divergent opinions Respects Chatham House Rules 21

5. Supporting Action Two examples of ways to support action Take an integrated approach to supporting evidence- and values-informed policymaking by convening a citizen panel followed by a stakeholder dialogue on the same topic informed by an evidence brief that includes findings from the citizen panel ‘off-the-record’ deliberations with those who are in a position to champion change Personalized briefings to those who can take action 22

Evaluation & Impact (1) Citizen brief (n=14 briefs; n=211 respondents) Mean overall assessment = 6.1 (SD = 1.2) Range of ratings of 14 features of citizen briefs = 5.4-6.4 Citizen panels (n=20 panels n=215 respondents) Mean overall assessment = 6.7 (SD = 1.2) Range of ratings of 11 specific design features = 6.3-6.9 Self-rated knowledge of the topic increased from pre- to post-panel (mean of 5.1 to 5.6) Example of impact Directly informed the Ontario Medical Association’s end-of-life care strategy and the Canadian Medical Association’s national dialogue about end-of-life care 23

Evaluation & Impact (2) Evidence briefs (n=41, n= 610 respondents) Overall rating of brief = 6.2, all but five features rated ≥ 6.0, and only three features had much variation (SD>1.1) Dialogues (n=41, n=613 respondents) Overall rating of dialogue = 6.2, all but one feature rated ≥ 6.1, and no features had much variation (SD>1.1) Examples of impact Directly informed an Ontario provincial cabinet submission about creating community-based specialty clinics Spurred formation of committee reporting to Ontario deputy minister to optimize clinical practice based on data, evidence and guidelines 24

Evaluation & Impact (3) Dialogues (continued) Ratings Strong behavioural intention to act and positive attitudes, but lower ratings of (and greater variability in) subjective norms and behavioural control Behavioural intentions I expect to use 6.2 [0.8] I want to use I intend to use 6.1 [0.8] Positive attitudes 6.3 [0.9] Subjective norms 5.7 [1.3] Perceived behavioural control 5.5 [1.5] 25

Training/Capacity Building Policymakers, stakeholders and researchers require support to develop skills in finding and using research evidence to inform the stages of policy development (clarifying problems, framing options, identifying implementation considerations) E.g., - Health Systems Learning Finding and Using Research Evidence to Inform Decision-making in Health Systems and Organizations Three course objectives: To develop knowledge about tools and resources available to help health system decision-makers in order to support their use of research evidence To examine the attitudes that are supportive of using research evidence in health system decision-making To enhance skills in acquiring, assessing, adapting and applying research evidence 26

Evaluation & Impact Training/capacity building 1,055 trainees (639 in-person, 190 in-person+online and 226 online) 531 policymakers, 42 managers, 47 students, 15 professionals, 16 researchers, 13 ‘other’ and 180 in multiple roles completed evaluation (RR=80%) Strongly positive feedback from evaluations (measured on seven-point Likert scale) Overall rating = 6.0 (range = 4 - 7) Highest rated feature: Material relevant to my professional development = 6.6 (range = 3-7) Lowest ratings: The workshop enhanced local applicability assessment skills = 5.8 (range = 2-7) 16 of 18 design features had average ratings of 6 or more 27

Facebook McMaster Heath Forum Stay current with updates about Forum-led initiatives, events and training opportunities, via Twitter @MacHealthForum; and Facebook McMaster Heath Forum 28