1. Pragmatic Trial Case Example
Immunization Delivery among Preschool Children
Pragmatic TRIALS

2. Motivation:
discussing A CASE Provides the CHANCE TO PRACTICE AND APPLY THE PRINCIPLES OF PRAGMATIC CLINICAL TRIALS.

3. Process Review background & research question
In small groups (2-3 people), discuss/brainstorm answers to the questions posed for each workshop module.
Each module will be debriefed before moving on to the next. We will report out possible answers.
A written debrief and case reference will be provided as a resource at the end of the day.

4. Goal Type of Intervention
Increase immunization delivery among preschool children
Type of Intervention
Clinical practice + public health
Registry-driven intervention and evaluation

5. Context
Vaccination is recognized as one of the greatest public health achievements of the 20th century.
However, only 44% of children aged months receive all recommended vaccines.
A Healthy People 2020 goal is to increase the proportion of children aged months who receive all recommended doses of childhood vaccines to 80%.

6. Context
The Community Preventive Services Task Force recommends reminder/recall to ↑ immunization rates.
Reminder/Recall = Notification for upcoming immunizations (reminders) and recall notices for overdue immunizations (recall).
Practice-based reminder/recall has been proven efficacious; however, few providers are conducting reminder/recall.
An alternative public-health based approach (working with regional or state immunization information systems) can facilitate reminder/recall using a centralized process.

7. Research Question:
Which is more effective at increasing immunizations among preschool children?
Practice-based reminder/recall; or
Population-based reminder/recall conducted centrally by health departments using a regional or state immunization information system

8. Design a pragmatic research trial to answer the research question.

9. Remember . . .
PRECIS-2 is a framework to support systematic decision-making in the design and reporting of pragmatic clinical trials.
There are no right or wrong answers.

10. Case Study Application
engagement

11. Discussion – 10 minutes
Q1: Which stakeholders (individuals and/or organization) would you engage?
Q2: When in the research process, and how, would you engage these stakeholders?

12. Report Out
Q1: Which stakeholders (individuals and/or organization) would you engage?

13. Possible stakeholders (the 7 P’s)
Concannon et al JGIM (2012)
Patients and the Public
Caregivers and families of young children, and patient advocacy groups
Providers
Individuals (e.g., physicians, nurses, pharmacists) and organizations (e.g., private practices, clinics, community health centers, pharmacies) that deliver vaccines to young children
Public Health
Organizations who are responsible for vaccine delivery and immunization surveillance (e.g., state and local public health departments
Purchasers
Employers, the self-insured, taxpayers and other entities responsible for underwriting the costs of vaccine delivery
Payers
Insurers, Medicaid, state insurance exchanges, individuals with deductibles, and others responsible for reimbursement of vaccines
Policy makers
Colorado Department of Public Health, professional associations, Colorado General Assembly (legislators), and the Governor’s office (or analogous policymakers in other states or at the national/federal level)
Product makers
Electronic health record and related technology companies; vaccine manufacturers
Principal investigators
Other researchers investigating reminder/recall interventions

14. Report Out
Q2: When in the research process, and how, would you engage these stakeholders?

15. Throughout the research process!
Study design
Study execution
Dissemination of the findings
See: PCORI Engagement Rubric

16. Study population & setting
Case Study Application
Study population & setting

17. Discussion – 20 minutes
Q1: ELIGIBILITY: What criteria will you use to select participants for your study? Generalizability?
Q2: RECRUITMENT: What type of approach will you use to recruit study participants? How much effort?
Q3: SETTING: Where should the trial be conducted? What criteria will you use to select sites? Generalizability?
Q4: ORGANIZATION: What expertise and resources will be required to deliver the intervention? Generalizability?

18. Report Out
Q1: ELIGIBILITY: What criteria will you use to select participants for your study? Generalizability?
Q2: RECRUITMENT: What type of approach will you use to recruit study participants? How much effort?

19. A two-level approach might be used.
ELIGIBILITY:
A two-level approach might be used.
Select counties:
No on-going reminder/recall efforts
High CIIS saturation levels
Similar healthcare delivery systems
Include all clinical practices that offer primary care visits and immunizations to young children.

20. RECRUITMENT: Possible Pragmatic Approach:
Invite all clinical practices in the selected counties to participate.
Offer training and provide minimal financial incentives for participation (e.g., only to off- set costs of reminder/recall mailings)

21. Report Out
Q3: SETTING: Where should the trial be conducted? What criteria will you use to select sites? Generalizability?
Q4: ORGANIZATION: What expertise and resources will be required to deliver the intervention? Generalizability?

22. SETTING: Possible Pragmatic Settings:
State/local public health departments.
Clinical practices that offer primary care visits and immunizations to children aged months.

23. ORGANIZATION: Possible level of expertise and resources:
Ability to identify target patients for reminder/recall at the population-level (public health department)
Ability to attend a training webinar (clinical practices) – R/R implementation would occur within the context of usual practices

24. Case Study Application
Research design

25. Discussion – 20 minutes
Q1: DESIGN: What trial design will you use? Why?
Q2: OUTCOME: What measures are relevant for your study participants? For other stakeholders who will be using the results of the study?
Q3: FOLLOW-UP: How often will you collect data? How similar is this approach to usual care? What data source(s) will you use? Are they valid and reliable?
Q4: PRIMARY ANALYSIS: Will you use an ITT or a per-protocol analysis? Why?

26. Report Out Q1: DESIGN: What trial design will you use?
Q2: OUTCOME: What measures are relevant for your study participants? For other stakeholders who will be using the results of the study?

27. DESIGN: Possible pragmatic trial designs: Cluster-randomized
(patients-practices-county)
Step-wedge likely not necessary as all counties will receive an intervention.

28. PRIMARY OUTCOME:
Possible outcomes relevant for participants and stakeholders:
% of children who are up-to-date using the national definition from the Advisory Committee on Immunization Practices
cost-effectiveness (% UTD per $$ spent)

29. Report Out
Q3: FOLLOW-UP: How often will you collect data? How similar is this approach to usual care? What data source(s) will you use? Are they valid and reliable?
Q4: PRIMARY ANALYSIS: Will you use an ITT or a per-protocol analysis?

30. FOLLOW-UP:
Possible Pragmatic Data Sources and Intensity of Measurement:
Colorado Immunization Information System (CIIS) -- ‘usual care’ for tracking immunizations
CIIS is a dynamic, real-time registry

31. PRIMARY ANALYSIS: Possible Statistical Approach:
ITT – UTD primary outcome (population-based)
Per protocol – cost effectiveness (for practices that implemented R/R)

32. Case Study Application
Real-world use

33. Discussion – 20 minutes
Q1: FLEXIBILITY (delivery): To what degree will you monitor and enforce intervention fidelity? How would this compare with flexibility allowed in usual care?
Q2: FLEXIBILITY (adherence): To what degree will you monitor and encourage adherence to the intervention by study participants? How would this compare with practices employed in usual care?

34. Report Out
Q1: FLEXIBILITY (delivery): To what degree will you monitor and enforce intervention fidelity? How would this compare with flexibility allowed in usual care?

35. FLEXIBILITY (delivery):
Possible approaches:
More pragmatic: clinical practice R/R, suggested approach via training.
More explanatory: population-based R/R, follow a specific centralized protocol

36. Report Out
Q2: FLEXIBILITY (adherence): To what degree will you monitor and encourage adherence to the intervention by study participants? How would this compare with practices employed in usual care?

37. FLEXIBILITY (adherence):
Possible approaches:
More pragmatic: clinical practices choose whether to implement R/R or not
More explanatory: population-based R/R overseen by the research team to ensure compliance

38. Thank you for your active participation – hopefully, a helpful learning experience

39. Key Take-Away’s
PRECIS-2 is a framework to support systematic decision-making in the design and reporting of pragmatic clinical trials.
Trial design decisions fall on a continuum between explanatory and pragmatic choices.
Trial design decisions should be guided by knowledge of how the results will be used and for what types of decisions. Pragmatic trials seek to inform decisions about practice and policy.

40. Case Source
Kempe A, Saville A, Dickinson LM, Eisert S, Reynolds J, Herrero D, Beaty B, Albright K, Dibert E, Koehler V, Lockhart, Calonge N. Population-Based Versus Recall for Childhood Immunizations: A Randomized Controlled Comparative Effectiveness Trial. Am J Public Health 2013; 103:

41. Population-based vs Practice-based Reminder/Recall: a Pragmatic Comparative Effectiveness Trial
Allison Kempe, MD, MPH

42. Background
Reminder/recall (R/R): postcards, letters or telephone calls to inform patients they are due or overdue for immunizations
Can be automated using Immunization Information System (IIS)
R/R conducted in practice settings shown effective in increasing rates but only 16% of physicians nationally are conducting
Population-based R/R if conducted centrally by public health departments could offer advantages of:
Reducing burden of conducting R/R by practices
Reaching children without usual source of primary care

43. Objectives To compare …
effectiveness and cost-effectiveness of conducting R/R using two methodologies:
Population-based R/R: conducted centrally by the State Health Department using the Colorado Immunization Information System (CIIS)
Practice-based R/R: conducted at the level of the primary care practice using CIIS

44. Methods: Randomization of Counties
14 Colorado Counties
6 Urban counties with similar income, race-ethnicity, population & CIIS saturation
3 counties
practice-based
R/R
3 counties population-based R/R
8 Rural counties with similar income, race-ethnicity, population & CIIS saturation
4 counties practice-based R/R
4 counties population-based R/R

45. Study Populations for Both Intervention Arms
Downloaded names and addresses of children months old needing 1 immunization within all 14 counties
Colorado Immunization Information System (CIIS)
Patient names, addresses and immunization data automatically uploaded from Birth Vital Statistics to

46. Methods: Intervention Strategies
Population-based recall counties:
Centralized R/R conducted by the State Public Health Department June – September 2010
Up to 3 mailings to children months needing immunizations
R/R notices suggested patients go to primary care provider for immunization or, if they did not have one, to public health immunization site

47. Methods: Intervention Strategies
Practice-based recall counties:
All practices invited to attend web-based R/R training in May/June 2010
R/R methodology suggested
3 mailings to children months needing immunizations
June – September 2010
Financial support for mailings offered to practices who did R/R in this timeframe

48. Methods: Statistical Analysis
To account for clustered nature of the data mixed effects models used
Two models conducted to assess association between intervention group and whether or not 1) child became UTD or 2) received any shot during the study period
Fixed effects for both models included county baseline UTD rate, rural/urban status of county, and whether or not site of last service did R/R
The random effect in both models was site of last service

49. Methods: Cost Assessment
Population-based R/R (performed centrally)
Staff time for training and implementation
Staff time for updating bad mailing addresses
Mailing and printing costs for up to 3 mailings
Practice-based R/R (performed differently at each practice)
Average staff time among practices conducting R/R
Average mailing costs or costs of phone calls

50. Comparison of “Reach”
of the Intervention

51. Percent Brought Up-to-Date within 6 months (of those needing vaccines at baseline)
We need stats here—brenda is doing
Would prefer 3-d bars for all slides

52. Subgroup Analysis within
Practice-based Counties Percent Brought Up-to-Date R/R vs no R/R
Need second slide with the bar on the nest slide in green in back of and slightly to the right of bar on left—needs to be clear that it is separate and needs to be labeled population-based…also there needs to be a p value for the difference between 24 and 19
n = 887
n = 17848

53. Subgroup Analysis w/in Practice-based Counties Percent Brought Up-to-Date R/R vs no R/R

54. Results: Multivariable Models
Association of Intervention Group with Two Outcomes
Outcomes Modeled
Adjusted OR (95% CI)
P-value
Becoming up-to-date in population-based versus practice-based county
1.24 ( )
.0002
Receiving any vaccine in population-based versus practice-based county
1.27 ( )
<.0001
Other variables included in the model were baseline county UTD rate, rural/urban status of county, site of last service and whether or not site of last service did R/R, all of which were not statistically significant

55. Cost of Conducting R/R per Practice

56. Cost of R/R Per Child who Received ≥1 Vaccine

57. Cost of R/R Per Child Brought Up-to-Date

58. Limitations
Population impossible to accurately denominate in all counties—but same method of approximation used in both intervention arms
Population-based R/R hampered by many inaccurate addresses from vital statistics
Practices may have conducted R/R after the 6 month period of F/U despite incentives
Costs were based on personnel report, rather than direct observation

59. Conclusions
Both practice-based and population-based R/R effective—practice-based slightly more effective when practices participated
Overall, at a county level population-based R/R was more effective than practice-based R/R because of lack of participation of practices even when incentives provided
Costs per practice or per child vaccinated were much lower for population-based R/R

60. Implications
Centralized population-based R/R conducted by Public Health Departments may be more effective and less costly alternative to practice-based R/R
Optimal approach might involve collaboration between practices and public health
R/R notices could appear to come from practice and public health department
Could be less costly if practices helped in updating of addresses
More information needed regarding acceptability from practices’ and patients’ perspectives

61. University of Colorado Denver
Study Team
Principal Investigator – Allison Kempe, MD, MPH
Karen Albright, PhD
Eva Dibert, MHA
Vicky Koehler, MPH
Alison Saville, MSPH, MSW
L. Miriam Dickinson, PhD
Brenda Beaty, MSPH
Sheri Eisert, PhD
Ned Calonge, MD
Joni Reynolds, RN, MSN
CDPHE & CIIS Collaborators
Diana Herrero, MS

62. Funding
Study supported by a Challenge Grant from the National Institutes of Health (Award Number RC1LM01513)
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Library of Medicine or the National Institutes of Health.