The Use of Immunization Information Systems in Vaccine Effectiveness Studies 43rd National Immunization Conference Leila C. Sahni Center for Vaccine Awareness and Research, Texas Children’s Hospital March 31, 2009

*No financial disclosures or conflicts of interest Contributors Texas Children’s Hospital Julie A. Boom, M.D.* Carol J. Baker, M.D.* Marcia A. Rench, RN, BSN* Leila C. Sahni* Centers for Disease Control and Prevention Jacqueline E. Tate, PhD* Manish M. Patel, M.D.* Umesh D. Parashar, M.D.* The research that I will be presenting today is the result of a grant from the CDC that was given to the City of Houston and then over to TCH. We are very grateful for the opportunity to work with our colleagues at the CDC who made this project possible. *No financial disclosures or conflicts of interest

Center for Vaccine Awareness and Research Objectives To validate immunization information obtained from an immunization information system (IIS) against parent and provider records; To assess the utility of an IIS in evaluating post-licensure vaccine effectiveness (VE). To compare VE calculated using IIS data to VE calculated using parent and provider data. Center for Vaccine Awareness and Research

IISs IISs are comprehensive repositories of immunization data for children and adults within a specific geographic region. IISs can assist in medical decision-making, reminder recall, determining coverage levels, and identifying pockets of need. In 2007, 71% (16.5 million) of children < 6 years had ≥ 2 immunizations in an IIS. At the time that this study was conducted, the Houston-Harris County Immunization Registry (HHCIR) was operated by Texas Children’s Hospital. In 2008, this registry contained data for 84% of children less than 6 years of age residing in the Greater Houston area (unpublished data, August 2008). In early 2008, 62% of public and 51% of private providers reported immunizations administered to HHCIR (unpublished data, September, 2008). In December, 2008 HHCIR merged with ImmTrac, the statewide immunization registry CDC. Vaccines and Immunizations: What is IIS? http://www.cdc.gov/vaccines/programs/iis/what-iis.htm. CDC. 2007 IIS Annual Report. http://www.cdc.gov/vaccines/programs/IIS/rates/2007-child-map.htm

The History of IISs in VE Studies Averhoff et al (2001) assessed hepatitis A vaccination coverage using an immunization registry. Piedra et al (2007) tracked demographic information of study participants using TWICES (public health EMR). Mahon et al (2008) assessed the utility of the Boston IIS as a tool to study VE. Averhoff et al, 2001 Hepatitis A immunization was offered to 30,000 children aged 2 to 17 years from January, 1995 through December, 2000 as part of an effort to control an ongoing disease outbreak in Butte County, California. Averhoff et al used an immunization registry maintained by Butte County Health Department clinics to record doses of vaccine administered and assess first and second dose vaccination coverage among children who received hepatitis A vaccine as part of the study. Vaccine effectiveness was calculated as part of the study, using the incidence of hepatitis A among vaccinated children compared to incidence among unvaccinated children. Piedra et al, 2007 In a study comparing the effectiveness of trivalent live attenuated intranasal influenza vaccine to inactivated influenza vaccine, Piedra et al used the Texas Wide Integrated Client Encounter System (TWICES) to track the demographic information of vaccine recipients Vaccine effectiveness against febrile respiratory illness, pneumonia and influenza was calculated Mahon et al, 2008 Most recently, Mahon et al evaluated the Boston Immunization Information System as a potential tool for conducting VE studies 6 provider offices that manually entered data into the IIS and 7 sites that electronically uploaded data using an EMR participated in the study Patient charts were randomly selected from provider offices and DTaP, hepatitis B, MMR and varicella vaccine information were compared to a registry record, if one could be found More than 88% of selected records matched a corresponding registry record, with patients who were 11 or older when they first visited a selected site less likely to have a registry record than younger patients When a corresponding registry record was found, immunization data were compared to the provider record. The most common discrepancies observed were: Up-to-date status Number of immunizations received – most disagreement between provider and registry records were due to registry incompleteness Dates immunizations were administered Formulations of vaccine administered – DTaP v. DTP v. DT The authors concluded that IISs could be used to conduct VE studies, but that registry data should be assessed and validated against provider data before such a study is conducted Averhoff et al. JAMA. 2001;286(23):2968-2973. Piedra et al. Pediatrics. 2007;120(3):e553-e564. Mahon et al. BMC. 2008;8:160 doi:10.1186/1471-2458-8-160.

Methods Part of a larger study that assessed post- licensure effectiveness of pentavalent rotavirus vaccine (RV5) during February – June, 2008. Parent/guardian survey assessing demographics, symptoms, duration, and severity of illness was administered. Fecal specimens were tested using an EIA (Premier Rotaclone®). During February through June, 2008, active rotavirus disease surveillance was conducted among emergency room patients and inpatients at Texas Children’s Hospital Children were enrolled using convenience sampling during evening and weekend hours. The emergency room triage log was used to identify any child with a chief complaint of vomiting, diarrhea, or fever. These children were then approached and assessed for eligibility. Children 15 days through 23 months of age with symptoms of diarrhea and/or vomiting were offered participation and consented for enrollment. Diarrhea was defined as the occurrence of three or more loose stools in a 24 hour period, while vomiting was defined as one or more episodes within a 24 hour period. Patients with symptoms for 11 or more days were ineligible for participation. Patients who were immunocompromised, who did not reside within the Greater Houston area, or whose parents did not speak either English or Spanish were excluded from participation. When available, fecal specimens were obtained from dirty diapers and tested for rotavirus Boom JA “Effectiveness of Pentavalent Rotavirus Vaccine in United States Clinical Practice.” 48th Annual ICAAC/IDSA 46th Annual Meeting, October 28, 2008, Washington, DC

Center for Vaccine Awareness and Research Immunization Records Requested from parent/guardian at enrollment Requested from up to 3 providers Obtained from the Houston-Harris County Immunization Registry (HHCIR) Participants with no immunization information from any source were excluded from analysis All doses of vaccine were entered into a study database and double-checked for accuracy For all case and control patients, immunization records were requested from several sources: A copy of the child’s immunization record was obtained from the parent during enrollment, if available. During the enrollment process, information about the child’s most recent immunization providers was requested. Information for up to 3 providers was obtained. These providers were contacted by study staff and a copy of the child’s immunization record was Study staff searched the local immunization registry for all enrolled patients, and immunization data were obtained for those who had an established record. A second registry query was conducted after study conclusion to capture immunization information for children whose records may have been entered after they were enrolled, as not all providers enter immunization data on a real-time basis. Patients for whom an immunization record could not be obtained from any source were excluded from analysis. Center for Vaccine Awareness and Research

Center for Vaccine Awareness and Research Statistical Analysis Doses of DTaP and PCV7 were compared to doses of RV5 across all data sources. Kappa statistics were calculated to determine the agreement between different sources of immunization records. We used descriptive statistics to determine the degree of similarity between data obtained from parent, provider, and registry records Pentavalent rotavirus vaccine, or RV5, was the vaccine of interest as the larger study aimed to assess the post-licensure effectiveness of RV5 RV5 data were compared to DTaP and PCV7 data. DTaP was selected as it represents a commonly-used comparison vaccine and is widely administered as part of the childhood immunization series. DTaP vaccine administered alone or in combination with other vaccines was considered valid. No distinction between DTaP alone or in combination (such as in Pediarix (DTaP-HepB-IPV) or Pentacel (DTaP-Hib-IPV) ) was made. Pneumococcal conjugate vaccine was selected primarily because it is the only childhood vaccine not currently available in combination with other vaccines. Center for Vaccine Awareness and Research

Sources of Immunization Data (All participants, n=628) Registry record n=284 Provider record n=555 Parent record n=131 n=226 n=208 n=58 n=64 n=6 n=16 n=3 No record from any source n=48 For all AGE and ARI patients (n=628) No record for 48 (8%) patients Total number of provider records: 555 (88% of all patients had provider record) Provider only: 225 (41% of provider records were provider record only) 36% of patients had provider record only Provider and parent: 64 (12% of provider records had corresponding parent record) 10% of patients had provider and parent records Provider and registry: 208 (38% of provider records had corresponding registry record) 33% of patients had provider and registry records Provider, parent, and registry: 58 (10%) 9% of patients had provider, parent, and registry records Total number of parent records: 131 (21% of all patients had parent record) Parent only: 6 (5% of parent records were parent record only ) 1% of patients had a parent record only Parent and provider: 64 (49% of parent records had a corresponding provider record) 10% of patients had parent and provider records Parent and registry: 3 (2% of parent records had a corresponding registry record) <1% of patients had parent and registry records Parent, provider, and registry: 58 (44% of parent records had a corresponding provider and registry record) 9% of patients had provider, parent, and registry records Total number of registry records: 285 (45% of all patients had a registry record) Registry only: 16 (6% of registry records were registry only) 3% of patients had registry record only Registry and provider: 208 (73% of registry records had corresponding provider record) 33% of patients had registry and provider record Registry and parent: 3 (1% of registry records had corresponding parent record) <1% of patients had registry and parent record Registry, parent, and provider: 58 (20% of registry records had corresponding parent and provider records) 9% of patients had parent, provider, and registry records

Center for Vaccine Awareness and Research DTaP and PCV7 Data DTaP* PCV7 Parent n=131 (21%) Provider n=555 (88%) Registry n=284 (45%) Combined n=580 (92%) Combined n=580 (92%) 0 doses† 5 (4) 38 (7) 27 (10) 40 (7) 4 (3) 36 (6) 1 dose 16 (12) 85 (15) 68 (24) 88 (15) 86 (15) 64 (23) 90 (16) 2 doses 22 (17) 82 (15) 40 (14) 80 (14) 24 (18) 89 (16) 49 (17) 83 (14) 3+ doses 88 (67) 350 (63) 149 (52) 372 (64) 87 (66) 344 (62) 144 (51) 369 (64) This table shows the number of doses of DTaP and PCV7 that each patient received through the date of their enrollment, by the three different sources (parent, provider, and registry), as well as the highest number of doses by the combined records As you can see, information for the two vaccines are very similar. *Number of doses of DTaP administered alone or in combination with other vaccine †0 doses of vaccine appropriate for some patients Center for Vaccine Awareness and Research

RV5 Data RV5 51 (39) 275 (50) 158 (56) 276 (48) 23 (18) 80 (14) Parent n=131 (21%) Provider n=555 (88%) Registry n=284 (45%) Combined n=580 (92%) 0 doses† 51 (39) 275 (50) 158 (56) 276 (48) 1 dose 23 (18) 80 (14) 45 (16) 91 (16) 2 doses 74 (13) 33 (12) 70 (12) 3+ doses 34 (26) 126 (23) 48 (17) 143 (25) †0 doses of vaccine appropriate for some patients

Center for Vaccine Awareness and Research RV5 and PCV7 Data RV5 PCV7 Combined n=580 (92%) Combined n=580 (92%) 0 doses† 276 (48) 38 (7) 1 dose 91 (16) 90 (16) 2 doses 70 (12) 83 (14) 3+ doses 143 (25) 369 (64) Here you can see a comparison of RV5 and PCV7 data †0 doses of vaccine appropriate for some patients Center for Vaccine Awareness and Research

Agreement Between Parent and Provider Records RV5 DTaP* PCV7 Records agree 111 (91) 108 (89) 107 (88) Parent doses> Provider doses 10 (8) 11 (9) Provider doses> Parent doses 1 (1) 3 (2) 4 (3) *Number of doses of DTaP administered alone or in combination with other vaccine

Agreement Between Parent and Registry Records RV5 DTaP* PCV7 Records agree 51 (84) 46 (75) Parent doses> Registry doses 9 (15) 14 (23) 15 (25) Registry doses> Parent doses 1 (2) 0 (0) *Number of doses of DTaP administered alone or in combination with other vaccine

Agreement Between Provider and Registry Records RV5 DTaP* PCV7 Records agree 206 (78) 187 (71) 186 (70) Provider doses> Registry doses 42 (16) 64 (24) Registry doses> Provider doses 17 (6) 14 (5) 15 (6) *Number of doses of DTaP administered alone or in combination with other vaccine

Agreement Between All Sources of Data Parent, Provider, and Registry n=58 RV5 DTaP* PCV7 Records agree 45 (78) 39 (67) 40 (69) Parent doses> Provider doses 5 (9) 6 (10) Provider doses> Parent doses 1 (2) 0 (0) Parent doses> Registry doses 9 (16) 14 (24) 15 (26) Registry doses> Parent doses Provider doses> Registry doses 12 (21) Registry Doses> Provider doses 4 (7) 3 (5) *Number of doses of DTaP administered alone or in combination with other vaccine

Agreement Between Different Data Sources Kappa Interpretation <0 No agreement 0.0-0.19 Poor agreement 0.20-0.39 Fair agreement 0.40-0.59 Moderate agreement 0.60-0.79 Substantial agreement 0.80-1.00 Almost perfect agreement The next slide shows the agreement between parent, provider, and registry immunization data for DTaP, PCV7 and RV5 using kappa statistics. As you can see, a kappa value of 0.60 or greater indicates substantial agreement between records, with a value of 0.80 or higher indicating almost perfect agreement Landis JR and Koch GG. The measurement of observer agreement for categorical data. Biometrics 22:159-174, 1977.

Agreement Between Different Data Sources RV5 PCV7* DTaP* к (95% CI) p-value Provider and Registry Record 265 0.65 subst. (0.58, 0.73) 0.002 0.50 mod. (0.42, 0.59) <0.001 0.49 mod. (0.41, 0.58) Parent and Registry Record 61 0.77 subst. (0.64, 0.90) 0.04 -- Provider and Parent Record 122 0.87 al. per. (0.80, 0.94) 0.03 0.76 subst. (0.65, 0.87) 0.27 0.78 subst. (0.67, 0.88) 0.08 For all three vaccines, agreement between parent and provider records is highest Looking only at RV5 data, the provider and registry and parent and registry records both had substantial agreement, with slightly more agreement between parent and registry records In all cases, RV5 data had more agreement across all data sources than either DTaP or PCV7 data. This may be because 48% of combined records had no doses of RV5 compared to only 7% of combined records for DTaP and PCV7. Agreement is much more likely to occur when there is no information. *DTaP and PCV comparing 0, 1, 2, and 3+ doses p-value from Bhapkar's test for marginal homogeneity

Availability of Immunization Data RV+ Cases ARI Controls RV- Controls n=90 (%) n=228 (%) p-value* n=115 (%) Parent record 18 (20) 39 (17) 0.55 29 (25) 0.38 Provider record 73 (81) 196 (86) 0.28 102 (89) 0.13 Registry record 44 (49) 96 (42) 0.27 55 (48) 0.88 Any record 79 (88) 206 (90) 0.5 108 (93) 0.20 *Compared to rotavirus positive cases

Vaccine Effectiveness RV- ARI HHCIR 3 doses 89% (95%: [70%, 96%]) 85% (95%: [55%, 95%]) 82% (95%: [19%, 96%]) As part of the larger study, vaccine effectiveness was calculated using rotavirus-negative AGE controls, concurrently enrolled ARI controls, and registry-selected controls. VE was similar for all 3 control groups – add values. Vaccine effectiveness was calculated using the formula (1 – odds ratio)*100% Center for Vaccine Awareness and Research

Conclusions When participant IIS data were available, they were similar to parent and provider records. Vaccine effectiveness calculated using IIS data was similar to parent and provider data. Due to the variability in quality and completeness of IIS data, validation prior to use in VE studies is advisable. Population-based IISs may represent an excellent source of immunization data. When participant data were found in the IIS, they were similar to parent and provider records. However, only 49% of rotavirus-positive case patients had a registry record.

Questions? Leila C. Sahni Texas Children’s Hospital lcsahni@texaschildrens.org (832) 824-2057 Center for Vaccine Awareness and Research