Pragmatic Trial Case Example Immunization Delivery among Preschool Children Pragmatic TRIALS WWW.Ucdenver.edu/implementation  

Motivation: discussing A CASE Provides the CHANCE TO PRACTICE AND APPLY THE PRINCIPLES OF PRAGMATIC CLINICAL TRIALS. WWW.Ucdenver.edu/implementation  

Process Review background & research question In small groups (2-3 people), discuss/brainstorm answers to the questions posed for each workshop module. Each module will be debriefed before moving on to the next. We will report out possible answers. A written debrief and case reference will be provided as a resource at the end of the day. WWW.Ucdenver.edu/implementation  

Goal Type of Intervention Increase immunization delivery among preschool children Type of Intervention Clinical practice + public health Registry-driven intervention and evaluation WWW.Ucdenver.edu/implementation  

Context Vaccination is recognized as one of the greatest public health achievements of the 20th century. However, only 44% of children aged 19-35 months receive all recommended vaccines. A Healthy People 2020 goal is to increase the proportion of children aged 19-35 months who receive all recommended doses of childhood vaccines to 80%.   WWW.Ucdenver.edu/implementation  

Context The Community Preventive Services Task Force recommends reminder/recall to ↑ immunization rates. Reminder/Recall = Notification for upcoming immunizations (reminders) and recall notices for overdue immunizations (recall). Practice-based reminder/recall has been proven efficacious; however, few providers are conducting reminder/recall. An alternative public-health based approach (working with regional or state immunization information systems) can facilitate reminder/recall using a centralized process. WWW.Ucdenver.edu/implementation  

Research Question: Which is more effective at increasing immunizations among preschool children? Practice-based reminder/recall; or Population-based reminder/recall conducted centrally by health departments using a regional or state immunization information system WWW.Ucdenver.edu/implementation  

Design a pragmatic research trial to answer the research question. WWW.Ucdenver.edu/implementation  

Remember . . . PRECIS-2 is a framework to support systematic decision-making in the design and reporting of pragmatic clinical trials. There are no right or wrong answers. WWW.Ucdenver.edu/implementation  

Case Study Application engagement WWW.Ucdenver.edu/implementation  

Discussion – 10 minutes Q1: Which stakeholders (individuals and/or organization) would you engage? Q2: When in the research process, and how, would you engage these stakeholders? WWW.Ucdenver.edu/implementation  

Report Out Q1: Which stakeholders (individuals and/or organization) would you engage? WWW.Ucdenver.edu/implementation  

Possible stakeholders (the 7 P’s) Concannon et al JGIM (2012) Patients and the Public Caregivers and families of young children, and patient advocacy groups Providers Individuals (e.g., physicians, nurses, pharmacists) and organizations (e.g., private practices, clinics, community health centers, pharmacies) that deliver vaccines to young children Public Health Organizations who are responsible for vaccine delivery and immunization surveillance (e.g., state and local public health departments Purchasers Employers, the self-insured, taxpayers and other entities responsible for underwriting the costs of vaccine delivery Payers Insurers, Medicaid, state insurance exchanges, individuals with deductibles, and others responsible for reimbursement of vaccines Policy makers Colorado Department of Public Health, professional associations, Colorado General Assembly (legislators), and the Governor’s office (or analogous policymakers in other states or at the national/federal level) Product makers Electronic health record and related technology companies; vaccine manufacturers Principal investigators Other researchers investigating reminder/recall interventions WWW.Ucdenver.edu/implementation  

Report Out Q2: When in the research process, and how, would you engage these stakeholders? WWW.Ucdenver.edu/implementation  

Throughout the research process! Study design Study execution Dissemination of the findings See: PCORI Engagement Rubric http://www.pcori.org/assets/2014/08/PCORI-Engagement-Rubric-with-Table.pdf WWW.Ucdenver.edu/implementation  

Study population & setting Case Study Application Study population & setting WWW.Ucdenver.edu/implementation  

Discussion – 20 minutes Q1: ELIGIBILITY: What criteria will you use to select participants for your study? Generalizability? Q2: RECRUITMENT: What type of approach will you use to recruit study participants? How much effort? Q3: SETTING: Where should the trial be conducted? What criteria will you use to select sites? Generalizability? Q4: ORGANIZATION: What expertise and resources will be required to deliver the intervention? Generalizability? WWW.Ucdenver.edu/implementation  

Report Out Q1: ELIGIBILITY: What criteria will you use to select participants for your study? Generalizability? Q2: RECRUITMENT: What type of approach will you use to recruit study participants? How much effort? WWW.Ucdenver.edu/implementation  

A two-level approach might be used. ELIGIBILITY: A two-level approach might be used. Select counties: No on-going reminder/recall efforts High CIIS saturation levels Similar healthcare delivery systems Include all clinical practices that offer primary care visits and immunizations to young children. WWW.Ucdenver.edu/implementation  

RECRUITMENT: Possible Pragmatic Approach: Invite all clinical practices in the selected counties to participate. Offer training and provide minimal financial incentives for participation (e.g., only to off- set costs of reminder/recall mailings) WWW.Ucdenver.edu/implementation  

Report Out Q3: SETTING: Where should the trial be conducted? What criteria will you use to select sites? Generalizability? Q4: ORGANIZATION: What expertise and resources will be required to deliver the intervention? Generalizability? WWW.Ucdenver.edu/implementation  

SETTING: Possible Pragmatic Settings: State/local public health departments. Clinical practices that offer primary care visits and immunizations to children aged 19-35 months. WWW.Ucdenver.edu/implementation  

ORGANIZATION: Possible level of expertise and resources: Ability to identify target patients for reminder/recall at the population-level (public health department) Ability to attend a training webinar (clinical practices) – R/R implementation would occur within the context of usual practices WWW.Ucdenver.edu/implementation  

Case Study Application Research design WWW.Ucdenver.edu/implementation  

Discussion – 20 minutes Q1: DESIGN: What trial design will you use? Why? Q2: OUTCOME: What measures are relevant for your study participants? For other stakeholders who will be using the results of the study? Q3: FOLLOW-UP: How often will you collect data? How similar is this approach to usual care? What data source(s) will you use? Are they valid and reliable? Q4: PRIMARY ANALYSIS: Will you use an ITT or a per-protocol analysis? Why? WWW.Ucdenver.edu/implementation  

Report Out Q1: DESIGN: What trial design will you use? Q2: OUTCOME: What measures are relevant for your study participants? For other stakeholders who will be using the results of the study? WWW.Ucdenver.edu/implementation  

DESIGN: Possible pragmatic trial designs: Cluster-randomized (patients-practices-county) Step-wedge likely not necessary as all counties will receive an intervention. WWW.Ucdenver.edu/implementation  

PRIMARY OUTCOME: Possible outcomes relevant for participants and stakeholders: % of children who are up-to-date using the national definition from the Advisory Committee on Immunization Practices cost-effectiveness (% UTD per $$ spent) WWW.Ucdenver.edu/implementation  

Report Out Q3: FOLLOW-UP: How often will you collect data? How similar is this approach to usual care? What data source(s) will you use? Are they valid and reliable? Q4: PRIMARY ANALYSIS: Will you use an ITT or a per-protocol analysis? WWW.Ucdenver.edu/implementation  

FOLLOW-UP: Possible Pragmatic Data Sources and Intensity of Measurement: Colorado Immunization Information System (CIIS) -- ‘usual care’ for tracking immunizations CIIS is a dynamic, real-time registry WWW.Ucdenver.edu/implementation  

PRIMARY ANALYSIS: Possible Statistical Approach: ITT – UTD primary outcome (population-based) Per protocol – cost effectiveness (for practices that implemented R/R) WWW.Ucdenver.edu/implementation  

Case Study Application Real-world use WWW.Ucdenver.edu/implementation  

Discussion – 20 minutes Q1: FLEXIBILITY (delivery): To what degree will you monitor and enforce intervention fidelity? How would this compare with flexibility allowed in usual care? Q2: FLEXIBILITY (adherence): To what degree will you monitor and encourage adherence to the intervention by study participants? How would this compare with practices employed in usual care? WWW.Ucdenver.edu/implementation  

Report Out Q1: FLEXIBILITY (delivery): To what degree will you monitor and enforce intervention fidelity? How would this compare with flexibility allowed in usual care? WWW.Ucdenver.edu/implementation  

FLEXIBILITY (delivery): Possible approaches: More pragmatic: clinical practice R/R, suggested approach via training. More explanatory: population-based R/R, follow a specific centralized protocol WWW.Ucdenver.edu/implementation  

Report Out Q2: FLEXIBILITY (adherence): To what degree will you monitor and encourage adherence to the intervention by study participants? How would this compare with practices employed in usual care? WWW.Ucdenver.edu/implementation  

FLEXIBILITY (adherence): Possible approaches: More pragmatic: clinical practices choose whether to implement R/R or not More explanatory: population-based R/R overseen by the research team to ensure compliance WWW.Ucdenver.edu/implementation  

Thank you for your active participation – hopefully, a helpful learning experience WWW.Ucdenver.edu/implementation  

Key Take-Away’s PRECIS-2 is a framework to support systematic decision-making in the design and reporting of pragmatic clinical trials. Trial design decisions fall on a continuum between explanatory and pragmatic choices. Trial design decisions should be guided by knowledge of how the results will be used and for what types of decisions. Pragmatic trials seek to inform decisions about practice and policy. WWW.Ucdenver.edu/implementation  

Case Source Kempe A, Saville A, Dickinson LM, Eisert S, Reynolds J, Herrero D, Beaty B, Albright K, Dibert E, Koehler V, Lockhart, Calonge N. Population-Based Versus Recall for Childhood Immunizations: A Randomized Controlled Comparative Effectiveness Trial. Am J Public Health 2013; 103:1116-1123. WWW.Ucdenver.edu/implementation  

Population-based vs Practice-based Reminder/Recall: a Pragmatic Comparative Effectiveness Trial Allison Kempe, MD, MPH WWW.Ucdenver.edu/implementation  

Background Reminder/recall (R/R): postcards, letters or telephone calls to inform patients they are due or overdue for immunizations Can be automated using Immunization Information System (IIS) R/R conducted in practice settings shown effective in increasing rates but only 16% of physicians nationally are conducting Population-based R/R if conducted centrally by public health departments could offer advantages of: Reducing burden of conducting R/R by practices Reaching children without usual source of primary care

Objectives To compare … effectiveness and cost-effectiveness of conducting R/R using two methodologies: Population-based R/R: conducted centrally by the State Health Department using the Colorado Immunization Information System (CIIS) Practice-based R/R: conducted at the level of the primary care practice using CIIS

Methods: Randomization of Counties 14 Colorado Counties 6 Urban counties with similar income, race-ethnicity, population & CIIS saturation 3 counties practice-based R/R 3 counties population-based R/R 8 Rural counties with similar income, race-ethnicity, population & CIIS saturation 4 counties practice-based R/R 4 counties population-based R/R

Study Populations for Both Intervention Arms Downloaded names and addresses of children 19-35 months old needing 1 immunization within all 14 counties Colorado Immunization Information System (CIIS) Patient names, addresses and immunization data automatically uploaded from Birth Vital Statistics to

Methods: Intervention Strategies Population-based recall counties: Centralized R/R conducted by the State Public Health Department June – September 2010 Up to 3 mailings to children 19-35 months needing immunizations R/R notices suggested patients go to primary care provider for immunization or, if they did not have one, to public health immunization site

Methods: Intervention Strategies Practice-based recall counties: All practices invited to attend web-based R/R training in May/June 2010 R/R methodology suggested 3 mailings to children 19-35 months needing immunizations June – September 2010 Financial support for mailings offered to practices who did R/R in this timeframe

Methods: Statistical Analysis To account for clustered nature of the data mixed effects models used Two models conducted to assess association between intervention group and whether or not 1) child became UTD or 2) received any shot during the study period Fixed effects for both models included county baseline UTD rate, rural/urban status of county, and whether or not site of last service did R/R The random effect in both models was site of last service

Methods: Cost Assessment Population-based R/R (performed centrally) Staff time for training and implementation Staff time for updating bad mailing addresses Mailing and printing costs for up to 3 mailings Practice-based R/R (performed differently at each practice) Average staff time among practices conducting R/R Average mailing costs or costs of phone calls

Comparison of “Reach” of the Intervention

Percent Brought Up-to-Date within 6 months (of those needing vaccines at baseline) We need stats here—brenda is doing Would prefer 3-d bars for all slides

Subgroup Analysis within Practice-based Counties Percent Brought Up-to-Date R/R vs no R/R Need second slide with the bar on the nest slide in green in back of and slightly to the right of bar on left—needs to be clear that it is separate and needs to be labeled population-based…also there needs to be a p value for the difference between 24 and 19 n = 887 n = 17848

Subgroup Analysis w/in Practice-based Counties Percent Brought Up-to-Date R/R vs no R/R

Results: Multivariable Models Association of Intervention Group with Two Outcomes Outcomes Modeled Adjusted OR (95% CI) P-value Becoming up-to-date in population-based versus practice-based county 1.24 (1.11-1.38) .0002 Receiving any vaccine in population-based versus practice-based county 1.27 (1.15-1.39) <.0001 Other variables included in the model were baseline county UTD rate, rural/urban status of county, site of last service and whether or not site of last service did R/R, all of which were not statistically significant

Cost of Conducting R/R per Practice

Cost of R/R Per Child who Received ≥1 Vaccine

Cost of R/R Per Child Brought Up-to-Date

Limitations Population impossible to accurately denominate in all counties—but same method of approximation used in both intervention arms Population-based R/R hampered by many inaccurate addresses from vital statistics Practices may have conducted R/R after the 6 month period of F/U despite incentives Costs were based on personnel report, rather than direct observation

Conclusions Both practice-based and population-based R/R effective—practice-based slightly more effective when practices participated Overall, at a county level population-based R/R was more effective than practice-based R/R because of lack of participation of practices even when incentives provided Costs per practice or per child vaccinated were much lower for population-based R/R

Implications Centralized population-based R/R conducted by Public Health Departments may be more effective and less costly alternative to practice-based R/R Optimal approach might involve collaboration between practices and public health R/R notices could appear to come from practice and public health department Could be less costly if practices helped in updating of addresses More information needed regarding acceptability from practices’ and patients’ perspectives

University of Colorado Denver Study Team Principal Investigator – Allison Kempe, MD, MPH Karen Albright, PhD Eva Dibert, MHA Vicky Koehler, MPH Alison Saville, MSPH, MSW L. Miriam Dickinson, PhD Brenda Beaty, MSPH Sheri Eisert, PhD Ned Calonge, MD Joni Reynolds, RN, MSN CDPHE & CIIS Collaborators Diana Herrero, MS

Funding Study supported by a Challenge Grant from the National Institutes of Health (Award Number RC1LM01513) The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Library of Medicine or the National Institutes of Health.