1. The Use of Immunization Information Systems in Vaccine Effectiveness Studies
43rd National Immunization Conference
Leila C. Sahni
Center for Vaccine Awareness and Research, Texas Children’s Hospital
March 31, 2009

2. *No financial disclosures or conflicts of interest
Contributors
Texas Children’s Hospital
Julie A. Boom, M.D.*
Carol J. Baker, M.D.*
Marcia A. Rench, RN, BSN*
Leila C. Sahni*
Centers for Disease Control and Prevention
Jacqueline E. Tate, PhD*
Manish M. Patel, M.D.*
Umesh D. Parashar, M.D.*
The research that I will be presenting today is the result of a grant from the CDC that was given to the City of Houston and then over to TCH. We are very grateful for the opportunity to work with our colleagues at the CDC who made this project possible.
*No financial disclosures or conflicts of interest

3. Center for Vaccine Awareness and Research
Objectives
To validate immunization information obtained from an immunization information system (IIS) against parent and provider records;
To assess the utility of an IIS in evaluating post-licensure vaccine effectiveness (VE).
To compare VE calculated using IIS data to VE calculated using parent and provider data.
Center for Vaccine Awareness and Research

4. IISs
IISs are comprehensive repositories of immunization data for children and adults within a specific geographic region.
IISs can assist in medical decision-making, reminder recall, determining coverage levels, and identifying pockets of need.
In 2007, 71% (16.5 million) of children < 6 years had ≥ 2 immunizations in an IIS.
At the time that this study was conducted, the Houston-Harris County Immunization Registry (HHCIR) was operated by Texas Children’s Hospital. In 2008, this registry contained data for 84% of children less than 6 years of age residing in the Greater Houston area (unpublished data, August 2008).
In early 2008, 62% of public and 51% of private providers reported immunizations administered to HHCIR (unpublished data, September, 2008).
In December, 2008 HHCIR merged with ImmTrac, the statewide immunization registry
CDC. Vaccines and Immunizations: What is IIS?
CDC IIS Annual Report.

5. The History of IISs in VE Studies
Averhoff et al (2001) assessed hepatitis A vaccination coverage using an immunization registry.
Piedra et al (2007) tracked demographic information of study participants using TWICES (public health EMR).
Mahon et al (2008) assessed the utility of the Boston IIS as a tool to study VE.
Averhoff et al, 2001
Hepatitis A immunization was offered to 30,000 children aged 2 to 17 years from January, 1995 through December, 2000 as part of an effort to control an ongoing disease outbreak in Butte County, California.
Averhoff et al used an immunization registry maintained by Butte County Health Department clinics to record doses of vaccine administered and assess first and second dose vaccination coverage among children who received hepatitis A vaccine as part of the study.
Vaccine effectiveness was calculated as part of the study, using the incidence of hepatitis A among vaccinated children compared to incidence among unvaccinated children.
Piedra et al, 2007
In a study comparing the effectiveness of trivalent live attenuated intranasal influenza vaccine to inactivated influenza vaccine, Piedra et al used the Texas Wide Integrated Client Encounter System (TWICES) to track the demographic information of vaccine recipients
Vaccine effectiveness against febrile respiratory illness, pneumonia and influenza was calculated
Mahon et al, 2008
Most recently, Mahon et al evaluated the Boston Immunization Information System as a potential tool for conducting VE studies
6 provider offices that manually entered data into the IIS and 7 sites that electronically uploaded data using an EMR participated in the study
Patient charts were randomly selected from provider offices and DTaP, hepatitis B, MMR and varicella vaccine information were compared to a registry record, if one could be found
More than 88% of selected records matched a corresponding registry record, with patients who were 11 or older when they first visited a selected site less likely to have a registry record than younger patients
When a corresponding registry record was found, immunization data were compared to the provider record. The most common discrepancies observed were:
Up-to-date status
Number of immunizations received – most disagreement between provider and registry records were due to registry incompleteness
Dates immunizations were administered
Formulations of vaccine administered – DTaP v. DTP v. DT
The authors concluded that IISs could be used to conduct VE studies, but that registry data should be assessed and validated against provider data before such a study is conducted
Averhoff et al. JAMA. 2001;286(23):
Piedra et al. Pediatrics. 2007;120(3):e553-e564.
Mahon et al. BMC. 2008;8:160 doi: /

6. Methods
Part of a larger study that assessed post- licensure effectiveness of pentavalent rotavirus vaccine (RV5) during February – June, 2008.
Parent/guardian survey assessing demographics, symptoms, duration, and severity of illness was administered.
Fecal specimens were tested using an EIA (Premier Rotaclone®).
During February through June, 2008, active rotavirus disease surveillance was conducted among emergency room patients and inpatients at Texas Children’s Hospital
Children were enrolled using convenience sampling during evening and weekend hours. The emergency room triage log was used to identify any child with a chief complaint of vomiting, diarrhea, or fever. These children were then approached and assessed for eligibility.
Children 15 days through 23 months of age with symptoms of diarrhea and/or vomiting were offered participation and consented for enrollment.
Diarrhea was defined as the occurrence of three or more loose stools in a 24 hour period, while vomiting was defined as one or more episodes within a 24 hour period.
Patients with symptoms for 11 or more days were ineligible for participation.
Patients who were immunocompromised, who did not reside within the Greater Houston area, or whose parents did not speak either English or Spanish were excluded from participation.
When available, fecal specimens were obtained from dirty diapers and tested for rotavirus
Boom JA “Effectiveness of Pentavalent Rotavirus Vaccine in United States Clinical Practice.” 48th Annual ICAAC/IDSA 46th Annual Meeting, October 28, 2008, Washington, DC

7. Center for Vaccine Awareness and Research
Immunization Records
Requested from parent/guardian at enrollment
Requested from up to 3 providers
Obtained from the Houston-Harris County Immunization Registry (HHCIR)
Participants with no immunization information from any source were excluded from analysis
All doses of vaccine were entered into a study database and double-checked for accuracy
For all case and control patients, immunization records were requested from several sources:
A copy of the child’s immunization record was obtained from the parent during enrollment, if available.
During the enrollment process, information about the child’s most recent immunization providers was requested. Information for up to 3 providers was obtained. These providers were contacted by study staff and a copy of the child’s immunization record was
Study staff searched the local immunization registry for all enrolled patients, and immunization data were obtained for those who had an established record.
A second registry query was conducted after study conclusion to capture immunization information for children whose records may have been entered after they were enrolled, as not all providers enter immunization data on a real-time basis.
Patients for whom an immunization record could not be obtained from any source were excluded from analysis.
Center for Vaccine Awareness and Research

8. Center for Vaccine Awareness and Research
Statistical Analysis
Doses of DTaP and PCV7 were compared to doses of RV5 across all data sources.
Kappa statistics were calculated to determine the agreement between different sources of immunization records.
We used descriptive statistics to determine the degree of similarity between data obtained from parent, provider, and registry records
Pentavalent rotavirus vaccine, or RV5, was the vaccine of interest as the larger study aimed to assess the post-licensure effectiveness of RV5
RV5 data were compared to DTaP and PCV7 data.
DTaP was selected as it represents a commonly-used comparison vaccine and is widely administered as part of the childhood immunization series. DTaP vaccine administered alone or in combination with other vaccines was considered valid. No distinction between DTaP alone or in combination (such as in Pediarix (DTaP-HepB-IPV) or Pentacel (DTaP-Hib-IPV) ) was made.
Pneumococcal conjugate vaccine was selected primarily because it is the only childhood vaccine not currently available in combination with other vaccines.
Center for Vaccine Awareness and Research

9. Sources of Immunization Data (All participants, n=628)
Registry record
n=284
Provider record
n=555
Parent record
n=131
n=226
n=208
n=58
n=64
n=6
n=16
n=3
No record from any source
n=48
For all AGE and ARI patients (n=628)
No record for 48 (8%) patients
Total number of provider records: 555 (88% of all patients had provider record)
Provider only: 225 (41% of provider records were provider record only) 36% of patients had provider record only
Provider and parent: 64 (12% of provider records had corresponding parent record) 10% of patients had provider and parent records
Provider and registry: 208 (38% of provider records had corresponding registry record) 33% of patients had provider and registry records
Provider, parent, and registry: 58 (10%) 9% of patients had provider, parent, and registry records
Total number of parent records: 131 (21% of all patients had parent record)
Parent only: 6 (5% of parent records were parent record only ) 1% of patients had a parent record only
Parent and provider: 64 (49% of parent records had a corresponding provider record) 10% of patients had parent and provider records
Parent and registry: 3 (2% of parent records had a corresponding registry record) <1% of patients had parent and registry records
Parent, provider, and registry: 58 (44% of parent records had a corresponding provider and registry record) 9% of patients had provider, parent, and registry records
Total number of registry records: 285 (45% of all patients had a registry record)
Registry only: 16 (6% of registry records were registry only) 3% of patients had registry record only
Registry and provider: 208 (73% of registry records had corresponding provider record) 33% of patients had registry and provider record
Registry and parent: 3 (1% of registry records had corresponding parent record) <1% of patients had registry and parent record
Registry, parent, and provider: 58 (20% of registry records had corresponding parent and provider records) 9% of patients had parent, provider, and registry records

10. Center for Vaccine Awareness and Research
DTaP and PCV7 Data
DTaP*
PCV7
Parent n=131 (21%)
Provider
n=555 (88%)
Registry
n=284 (45%)
Combined
n=580 (92%)
Combined n=580 (92%)
0 doses†
5 (4)
38 (7)
27 (10)
40 (7)
4 (3)
36 (6)
1 dose
16 (12)
85 (15)
68 (24)
88 (15)
86 (15)
64 (23)
90 (16)
2 doses
22 (17)
82 (15)
40 (14)
80 (14)
24 (18)
89 (16)
49 (17)
83 (14)
3+ doses
88 (67)
350 (63)
149 (52)
372 (64)
87 (66)
344 (62)
144 (51)
369 (64)
This table shows the number of doses of DTaP and PCV7 that each patient received through the date of their enrollment, by the three different sources (parent, provider, and registry), as well as the highest number of doses by the combined records
As you can see, information for the two vaccines are very similar.
*Number of doses of DTaP administered alone or in combination with other vaccine
†0 doses of vaccine appropriate for some patients
Center for Vaccine Awareness and Research

11. RV5 Data RV5 51 (39) 275 (50) 158 (56) 276 (48) 23 (18) 80 (14)
Parent n=131 (21%)
Provider
n=555 (88%)
Registry
n=284 (45%)
Combined
n=580 (92%)
0 doses†
51 (39)
275 (50)
158 (56)
276 (48)
1 dose
23 (18)
80 (14)
45 (16)
91 (16)
2 doses
74 (13)
33 (12)
70 (12)
3+ doses
34 (26)
126 (23)
48 (17)
143 (25)
†0 doses of vaccine appropriate for some patients

12. Center for Vaccine Awareness and Research
RV5 and PCV7 Data
RV5
PCV7
Combined
n=580 (92%)
Combined n=580 (92%)
0 doses†
276 (48)
38 (7)
1 dose
91 (16)
90 (16)
2 doses
70 (12)
83 (14)
3+ doses
143 (25)
369 (64)
Here you can see a comparison of RV5 and PCV7 data
†0 doses of vaccine appropriate for some patients
Center for Vaccine Awareness and Research

13. Agreement Between Parent and Provider Records
RV5
DTaP*
PCV7
Records agree
111 (91)
108 (89)
107 (88)
Parent doses> Provider doses
10 (8)
11 (9)
Provider doses> Parent doses
1 (1)
3 (2)
4 (3)
*Number of doses of DTaP administered alone or in combination with other vaccine

14. Agreement Between Parent and Registry Records
RV5
DTaP*
PCV7
Records agree
51 (84)
46 (75)
Parent doses> Registry doses
9 (15)
14 (23)
15 (25)
Registry doses> Parent doses
1 (2)
0 (0)
*Number of doses of DTaP administered alone or in combination with other vaccine

15. Agreement Between Provider and Registry Records
RV5
DTaP*
PCV7
Records agree
206 (78)
187 (71)
186 (70)
Provider doses> Registry doses
42 (16)
64 (24)
Registry doses> Provider doses
17 (6)
14 (5)
15 (6)
*Number of doses of DTaP administered alone or in combination with other vaccine

16. Agreement Between All Sources of Data
Parent, Provider, and Registry
n=58
RV5
DTaP*
PCV7
Records agree
45 (78)
39 (67)
40 (69)
Parent doses> Provider doses
5 (9)
6 (10)
Provider doses> Parent doses
1 (2)
0 (0)
Parent doses> Registry doses
9 (16)
14 (24)
15 (26)
Registry doses> Parent doses
Provider doses> Registry doses
12 (21)
Registry Doses> Provider doses
4 (7)
3 (5)
*Number of doses of DTaP administered alone or in combination with other vaccine

17. Agreement Between Different Data Sources
Kappa
Interpretation
<0
No agreement
Poor agreement
Fair agreement
Moderate agreement
Substantial agreement
Almost perfect agreement
The next slide shows the agreement between parent, provider, and registry immunization data for DTaP, PCV7 and RV5 using kappa statistics.
As you can see, a kappa value of 0.60 or greater indicates substantial agreement between records, with a value of 0.80 or higher indicating almost perfect agreement
Landis JR and Koch GG. The measurement of observer agreement for categorical data. Biometrics 22: , 1977.

18. Agreement Between Different Data Sources
RV5
PCV7*
DTaP*
к (95% CI)
p-value
Provider and Registry Record
265
0.65 subst. (0.58, 0.73)
0.002
0.50 mod. (0.42, 0.59)
<0.001
0.49 mod. (0.41, 0.58)
Parent and Registry Record 61
0.77 subst. (0.64, 0.90)
0.04 --
Provider and Parent Record
122
0.87 al. per. (0.80, 0.94)
0.03
0.76 subst. (0.65, 0.87)
0.27
0.78 subst. (0.67, 0.88)
0.08
For all three vaccines, agreement between parent and provider records is highest
Looking only at RV5 data, the provider and registry and parent and registry records both had substantial agreement, with slightly more agreement between parent and registry records
In all cases, RV5 data had more agreement across all data sources than either DTaP or PCV7 data.
This may be because 48% of combined records had no doses of RV5 compared to only 7% of combined records for DTaP and PCV7. Agreement is much more likely to occur when there is no information.
*DTaP and PCV comparing 0, 1, 2, and 3+ doses
p-value from Bhapkar's test for marginal homogeneity

19. Availability of Immunization Data
RV+ Cases
ARI Controls
RV- Controls
n=90 (%)
n=228 (%)
p-value*
n=115 (%)
Parent record
18 (20)
39 (17)
0.55
29 (25)
0.38
Provider record
73 (81)
196 (86)
0.28
102 (89)
0.13
Registry record
44 (49)
96 (42)
0.27
55 (48)
0.88
Any record
79 (88)
206 (90)
0.5
108 (93)
0.20
*Compared to rotavirus positive cases

20. Vaccine Effectiveness
RV-
ARI
HHCIR
3 doses
89%
(95%: [70%, 96%])
85%
(95%: [55%, 95%])
82%
(95%: [19%, 96%])
As part of the larger study, vaccine effectiveness was calculated using rotavirus-negative AGE controls, concurrently enrolled ARI controls, and registry-selected controls. VE was similar for all 3 control groups – add values.
Vaccine effectiveness was calculated using the formula (1 – odds ratio)*100%
Center for Vaccine Awareness and Research

21. Conclusions
When participant IIS data were available, they were similar to parent and provider records.
Vaccine effectiveness calculated using IIS data was similar to parent and provider data.
Due to the variability in quality and completeness of IIS data, validation prior to use in VE studies is advisable.
Population-based IISs may represent an excellent source of immunization data.
When participant data were found in the IIS, they were similar to parent and provider records.
However, only 49% of rotavirus-positive case patients had a registry record.

22. Questions? Leila C. Sahni Texas Children’s Hospital
(832)
Center for Vaccine Awareness and Research