1. Design Strategies
CRISP D&I Training Workshop 2013
Collaborative Population-based vs Practice-based Reminder/Recall to Increase Immunization Rates in Young Children A Pragmatic Comparative Effectiveness Trial
Allison Kempe, MD, MPH
Director Children’s Outcomes Research (COR) and Center for Research in Implementation Science and Prevention (CRISP)

2. Immunizations Second Only to Clean Water!
Disease
Pre-Vaccine Era Estimated Annual Morbidity*
Most Recent
Estimates‡of
U.S. Cases
Percent decrease
Diphtheria
21,053 0†
100%
H. influenzae (invasive, <5 years of age)
20,000
243†§
99%
Hepatitis A
117,333
11,049‡
91%
Hepatitis B (acute)
66,232
11,269‡
83%
Measles
530,217
61†
>99%
Mumps
162,344
982†
Pertussis
200,752
13,506†
93%
Pneumococcal disease (invasive, <5 years of age)
16,069
4,167‡
74%
Polio (paralytic)
16,316
Rubella
47,745 4†
Congenital Rubella Syndrome
152 1†
Smallpox
29,005
Tetanus
580
14†
98%
Varicella
4,085,120
449,363‡
89%
*CDC. JAMA, November 14, 2007; 298(18):2155–63
†CDC. MMWR, January 8, 2010; 58(51,52):1458–68
‡2008 estimates, S. pneumoniae estimates from Active Bacterial Core Surveillance
§25 type b and 218 unknown

3. Background – The Problem
Goal: Increase the proportion of children aged 19 to 35 months who receive all routinely recommended vaccines
Target = 80%
Estimated vaccination coverage for vaccination series among children months – National Immunization Survey, United States 2010
State/Area
Vaccine series modified
United States month olds
73% (27% not UTD)
Colorado month olds
68% (32% not UTD)
Can we move increase to left and increase font size of target 80%

4. WHY?! Barriers to optimal immunization delivery Financial
Access to care issues
Lack of awareness
Infrastructure and regulatory issues
Complexity and expansion of vaccination schedule
# of vaccines more than doubled in past 25 years
By18 months of age U.S. children recommended to receive vaccines against 14 different diseases, requiring up to 26 different vaccine doses
Vaccine hesitancy
Misinformation
Safety concerns

5. What Has Been Shown to Work?
Based on strong evidence of effectiveness Community Preventive Services Task Force recommends reminder/recall (R/R)
Notification to inform patients they are due (reminder) or overdue (recall) for immunizations
Can be delivered by mail, phone, autodialer, text,
In the past usually conducted at the level of the practice using practice-based data systems
47 RCTs included in recent Cochrane review with most showing effectiveness 5-15%....HOWEVER…

6. What Has Been Shown to Work?
35/47 trials were done by research teams coming into practices and none measured sustainability!
Our study team spent years studying and trying to improve practice-based R/R
RCTs to measure effectiveness in different settings and populations
Qualitative work to assess barriers to conducting R/R and sustainability of R/R

7. What Has Been Shown to Work?
What did we learn?
R/R difficult to implement even within highly motivated practices using an Immunization Information System (IIS)—lots of barriers to implementation and sustainability identified (AMONG OTHERS…)
Competing demands of primary care
Inadequate systems and inadequate training
Cost considerations
National data suggest that 6% of physicians are conducting practice-based R/R

8. Learning from Implementation Failure
Time for a new idea that accomplishes what practices want for their patients but facilitates implementation and sustainability!

9. Learning from Providers and Patients
What did providers want?
Would prefer to do R/R for their patients but majority think its not feasible
Vast majority (85%) were alright with Public Health Department (PHD) taking a major role
What did parents want?
Roughly half preferred to receive R/R from their child’s provider and half did not care if R/R came from provider or PHD or preferred PHD

10. Our Implementation Idea: Centralized and Collaborative
Make use of CIIS to simplify and centralize R/R
Algorithms to determine children not UTD at population level
Can generate mailing addresses, lists for generated autodialer or text messages
State PHD can conduct R/R centrally
Reduce burden of conducting R/R by practices
Reach children without usual source
R/R messages could come from practice and PHD
Responsive to provider and parent desires
If practices actively collaborated with PHD to update addesses this could decrease R/R costs care

11. Methods for Developing the Collaborative Intervention
Surveys of Parents and Providers
Qualitative data collection
Advisory Committees of parents and providers who reviewed qualitative information and made recommendations

12. Implementation Principles Behind Collaborative Centralized Approach
Make use of CIIS to simplify and centralize R/R
Algorithms to determine children not UTD at population level
Can generate mailing addresses, lists for generated autodialer or text messages
State PHD can conduct R/R centrally
Reduce burden of conducting R/R by practices
Reach children without usual source
cDesigning interventions for sustainability (low burden,
capitalizing on existing technologies)
Designing for scalability (populations)

13. Implementation Principles Behind Collaborative Centralized Approach
R/R messages could come from practice and PHD
Responsive to provider and parent desires
If practices actively collaborated with PHD to update addesses this could decrease R/R costs care
Intervention guided by preferences of providers and
patients
Endorsement by trusted professional/knowledge
transfer
Collaboration between stakeholders to increase
efficiency and decrease cost

14. Implementation Principles Behind Collaborative Centralized Approach
Advisory Committees of parents and providers who reviewed qualitative information and made recommendations for design of intervention
Qualitative data collection from both parents and providers (before, during and after intervention)
Surveys of parents and providers
Iterative testing with stakeholders

15. Agenda Setting
The goals were highly pragmatic—how to increase immunization rates for entire populations of children with highest level of efficiency, lowest cost and without undermining concept of medical home
Required highly collaborative approach

16. Developing Partnerships
Public Health Departments (PHDs)
State PHD was a primary partner in design and execution
Association of Local Public Health Organizations (CALPHO) consultant on our grant
Local PHDs—presented idea at statewide meeting
Input at meeting
Key informant interviews before and after intervention
CO Children’s Immunization Coalition (CCIC)
Local primary care provider organizations (AAP, AAFP) to select Provider Advisory Committee
Parent Advisory Committees selected with help from providers

17. Selecting the Population/Setting for Implementation
The population included all children in identified counties focus on heterogeneous populations consistent with pragmatic trial rather than classic clinical trial
Settings included both urban and rural counties and all primary care providers for children focus on heterogeneous settings and providers consistent with pragmatic trial rather than classic clinical trial

18. The Evaluation Framework: Re-AIM
Reach (proportion of the target population that participated in the intervention)
Effectiveness (success rate if implemented as in guidelines)
Adoption (proportion of settings, practices, and plans that will adopt this intervention)
Implementation (extent to which intervention is implemented as intended in the real world)
Maintenance (extent to which a program is sustained over time)

19. Primary Objective
To compare the effectiveness and cost effectiveness of conducting R/R to increase immunization rates in young children using two methodologies:
1. Collaborative Population-based R/R: conducted centrally by the state health department using the Colorado Immunization Information System (CIIS) with collaboration from private practices
2. Practice-based R/R: R/R conducted at the level of the primary care practice using CIIS

20. Comparison of “Reach” of Intervention

21. Percent Receiving Any Vaccine within 6 months (of those needing vaccines at baseline)
We need stats—brenda is doing

22. Percent Brought Up-to-Date within 6 months (of those needing vaccines at baseline)
We need stats here—brenda is doing
Would prefer 3-d bars for all slides

23. Subgroup Analysis w/in Practice-based Counties Percent Brought Up-to-Date R/R vs no R/R
Need second slide with the bar on the nest slide in green in back of and slightly to the right of bar on left—needs to be clear that it is separate and needs to be labeled population-based…also there needs to be a p value for the difference between 24 and 19
n = 887
n = 17848

24. Subgroup Analysis w/in Practice-based Counties Percent Brought Up-to-Date R/R vs no R/R

25. Cost of R/R Per Child who Received ≥1 Vaccine

26. Cost of R/R Per Child Brought Up-to-Date

27. Lessons Learned
Design implementation interventions with sustainability in mind from the get-go!
Know when to rethink the plan! (know when to hold ‘em, know when to fold ‘em….)
3. Listen to stakeholders and recipients of
intervention and continue to adjust iteratively

28. References
Kempe A, Saville A, Dickinson M, Reynolds J, Herrero D, Beaty B, Eisert S,  Albright K, Dibert E, Koehler V, Calonge N. Population-based versus Practice-based Recall for Childhood Immunizations:  A Randomized Controlled Comparative Effectiveness Trial. Am J Public Health Jun;103(6): doi: /AJPH PMID:
Albright K, Gechter K, Kampe A. Importance of mixed methods in pragmatic trials and dissemination and implementation research. Acad Peds 2013 Sep-Oct;13(5):400-7.
Suh CA, Saville A, Daley MF, Glazner JE, Barrow J, Stokley S, Dong F, Beaty B, Dickinson LM, Kempe A. Effectiveness and net cost of reminder/recall for adolescent immunizations. Pediatrics Jun;129(6):e Epub 2012 May 7.
Kempe A, Daley MF, Barrow J, Allred N, Hester N, Beaty BL, Crane LA, Pearson K, Berman S. Implementation of Universal Influenza Immunization Recommendations for Healthy Young Children: Results of a Randomized, Controlled Trial with Registry-based Recall. Pediatrics 2005 Jan;115(1):