Understanding IIS Performance and Dynamics 44th National Immunization Conference April 19-22, 2010 Atlanta, GA Understanding IIS Performance and Dynamics Exploring IIS data quality issues through IISAR - a national perspective Good Afternoon. My name is Vahé Heboyan and I am a consultant at the IIS Support Branch at the CDC. Along with my colleagues Janet Kelly and Bobby Rasulnia will be discussing how data from the IIS Annual Report can be used to enhance understanding of IIS performance and data quality issues. Vahé Heboyan Carter Consulting Inc. Janet Kelly CDC IISSB Bobby Rasulnia

Objectives Examine relationships between IISAR measures to understand data quality issues facing IIS. Discuss the extent IISAR data can be used for inferences. Determine if findings can offer ideas for future evaluation studies. The objectives are: To examine relationships between IISAR measures to understand data quality issues facing IIS; To discuss the extent IISAR data can be used for inferences; and To determine if findings can offer ideas for future evaluation studies. 44th National Immunization Conference | April 19-22, 2010 | Atlanta, GA

Data Source 2008 Immunization Information System Annual Report (IISAR) A survey of 64 grantees in 50 states, five cities, the District of Columbia, and 8 island territories that receive funding under the Public Health Service Act. Self-administered, Internet-based questionnaire. Data are self reported and might contain reporting bias. 88 measures Child, adolescent, and adult participation Provider site participation Vaccine series coverage Functional Standards Data quality, use, sharing 51 of 56 IIS grantees reported. Island territories were not included. Our data source is the 2008 IIS Annual Report. It is a survey of 64 grantees in 50 states, five cities, the District of Columbia, and 8 island territories that receive funding under the Public Health Service Act. Data is collected using a self-administered, Internet-based questionnaire. And since data are self reported they might contain reporting bias. The 2008 Annual Report contains 88 measures. However, due to the time constraints and specific objectives, we are only focusing on few measures that help to better understand data quality issues facing IIS. Data from 51 IIS are used in the analysis. Island territories and non-reporting grantees were not included in the data analysis. 44th National Immunization Conference | April 19-22, 2010 | Atlanta, GA

Methodology IISAR data were analyzed to: Make comparative assessment of IIS measures for grantee groups. Assess data quality issues, gaps, and challenges. Understand the extent performance in one IIS measure may characterize performance in another area. We have cross-examined and analyzed select IIS measures reported in the Annual Report to: Identify existence of relationships; Make comparative assessment of IIS measures for grantee groups; Assess data quality issues, gaps, and challenges; and Understand the extent performance in one IIS measure may characterize performance in another area. 44th National Immunization Conference | April 19-22, 2010 | Atlanta, GA

Does higher child participation result in higher Core Data Element completeness in IIS? NVAC Core Data Elements (CDE) are a crucial piece of information that have vital implication on data exchange, deduplication, vaccine inventory and recall mgmt, IZ status evaluation, RR, and data analysis. Thus, it is important to understand the factors that effect completeness of these elements. For this purpose, in the next couple of slides we will be looking at the CDE with an objective of utilizing IISAR data to understand completeness of the elements. We want to see if data would suggest substantial differences in CDE completeness due to level of child participation or initial data source. On this graph CDE completeness for IIS with more than 80% <6 years old child participation rate is illustrated in GREEN. RED shows CDE completeness for IIS with less than 80% child participation. As you can see from the chart, data suggest that there is NO substantial difference in core data element completeness associated with the level of child participation. Both, high and low child participation grantees exhibit similar patterns in populating CDE fields. We have also broken down the child participation into sub-categories (0-33; 34-66; 67-94; 95-100), and similar patterns emerged similar to the one on the screen. This highlights that CDE completeness does not differ as child participation rates change. 44th National Immunization Conference | April 19-22, 2010 | Atlanta, GA

Do population-based IIS have more complete core data elements? To further assess NVAC CDE completeness, we looked at the IIS based on their initial data source. Population based IIS establish newborn records by birth data from the birth records, such as vital records, birthing hospitals, perinatal or newborn screening programs and etc.). Currently, 82% (42/51) of IIS are population based. On this graph, CDE completeness for population based IIS is shown in GREEN. The RED illustrates CDE completeness for non-population based IIS. [ x 5] As we can see, these 2 groups of IIS do exhibit substantial differences in the completeness of several CDE as shown by BLUE circles. Which suggests, that there is a potential for increasing completeness of CDE by exploring the possibility of populating IIS by birth data. However, data also shows that even population based IIS have challenges in fully populating certain CDE fields. Thus, it becomes important to understand the underlying factors to be able to design effective and targeted interventions for fully populating all fields. 44th National Immunization Conference | April 19-22, 2010 | Atlanta, GA

Does higher child participation result in higher Mother’s Last Name completeness in IIS? - -  - - <6 years old child participation in IIS - -- - % populated Mother’s Last Name field Even though at national level IIS child participation did not show substantial difference in CDE completeness, we have looked at the individual CDE with a hypothesis that IIS with higher child participation may have higher CDE completeness. On the top graph <6 years old child participation is shown in GREEN for IIS with more than 80% participation rate. Completeness of Mother’s Last Name in IIS is shown in RED for this group. Same is illustrated for IIS with less than 80% child participation on the lower graph. On both graphs, IIS are ordered by ascending child participation. As it can be seen, there is no distinct pattern suggesting that achieving high child participation rates translates into higher completeness of MLN element in IIS. Similar conclusion was reached when looking at other CDE as well. Such as MFN, MMN, vaccine lot number, vaccine manufacturer, race, ethnicity, and birth state. 44th National Immunization Conference | April 19-22, 2010 | Atlanta, GA

Summary of IIS core data element completeness analysis No substantial difference in core data completeness associated with higher child participation rate. Populating IIS by birth records may result in higher core data element completeness in IIS. Achieving completeness of core data elements in IIS remains a challenge irrespective of IIS child participation rate and initial source for establishing record in IIS. For example, “Field not being required of providers” cited as a primary reason. Based on the analysis, we conclude that there is no substantial difference in core data completeness associated with higher child participation. This is true at the national aggregate level as well as when looking at the individual CDE across all IIS. However, IIS populated by birth data may lead to higher populated core data in IIS. Analysis show that regardless of child participation level and initial data source, systematic challenges exist for all IIS. For example, several IIS indicate that the field is not being required of providers as a primary reason for fields not being fully populated. 44th National Immunization Conference | April 19-22, 2010 | Atlanta, GA

Effect of birth records on timeliness of establishing an IIS record Populating IIS by birth data substantially increases timeliness of establishing an IIS record. Ensuring all records are established w/in 6 weeks remains a challenge. Average 5 weeks 81% w/in 6 weeks Establishment of newborn record in a timely manner is important for data exchange, IZ status assessment especially in early weeks and months, and timely RR. The top graph illustrates the timeliness of establishing a newborn record in IIS for those that are populated by birth data. Currently, 82% (42/51) IIS are population based. Timeliness is defined as being able to establish an IIS record within 6 weeks of birth. As it can be seen. Majority of IIS (57%) in this group establish 90-100% of newborn IIS records within the recommended 6 weeks. On average, IIS in this group establish a newborn record in 5 weeks with 81% of the records being established w/in recommended 6 week period. In contrast, as shown on the bottom graph, it takes in average 27 weeks to establish a newborn child record in non-population based IIS with only 38% of records being established w/in recommended 6 weeks. As you can see, populating IIS by birth data substantially increases the timeliness of establishing a record. The data also show that even the population based IIS have challenges to fully populating IIS w/in the recommended 6 weeks. Average 27 weeks 38% w/in 6 weeks 44th National Immunization Conference | April 19-22, 2010 | Atlanta, GA

Assessing contributing factors in series completeness in IIS Varicella does not appear to be the primary source of missing 1 dose Further evaluation studies are needed to identify missing dose(s) at vaccine level at each IIS . Use information to design targeted interventions to capture “low hanging fruit” to increase series completeness rates in IIS. IIS Annual Report data suggest that 18% and 16% of 19-35 mos. old children in the IIS are missing just one dose from being UTD on 4:3:1:3:3 and 4:3:1:3:3:1 vaccine series, respectively. This slide attempts to assess contributing factors in vaccine series completeness in IIS. Vaccine series completeness for 4:3:1:3:3 is shown in BLACK and for 4:3:1:3:3:1 in RED. As we can see both series are almost identical, thus suggesting that varicella does not appear to be the primary source of the missing dose among all IIS. Unfortunately IISAR data do not permit to examine series completeness for other antigens. However, data available at IIS can enable to identify missing dose(s) and thus design targeted interventions to capture the “low hanging fruit” and increase vaccine series completeness rates in IIS . Other studies have shown that the 4th DTaP is one of the primary doses missing from series coverage. IIS should conduct missing dose coverage assessments to determine the missing doses. Zhen Zhao et al. (Vaccine, 2009) Luman (JAMA, 2005) Kelly and Rasulnia (PAS poster, 2005) 44th National Immunization Conference | April 19-22, 2010 | Atlanta, GA

Next Steps Core Data Elements Timeliness Design evaluation studies to specially identify factors effecting core data completeness in IIS. Standardize guidance and baseline measures for core data elements at national level. Standardize for third-party objective review. Timeliness Address barriers to timely establishment of birth record. Immunization Assessment for Immunization Programs Identify and develop strategies to capture missing dose(s) . Use findings to develop targeted interventions to increase vaccine series completion rates in IIS. Based on our analysis, the following steps can be pursued to address data completeness and timeliness challenges in IIS. Evaluating core data element completeness factors. Standardize guidance and baseline measures for core data elements at national level. Assess barriers to timely establishment of newborn records. Develop targeted interventions to increase immunization coverage by assessing missing doses. 44th National Immunization Conference | April 19-22, 2010 | Atlanta, GA

Useful resources Contacts IIS Data Query Tool Vahé Heboyan, IBQ4@cdc.gov , 404-639-5245 IIS Data Query Tool http://www2a.cdc.gov/nip/registry/IISAR/IISAR_QUERY.asp IIS Publications Database http://www2a.cdc.gov/nip/IIS/IISPubs/IISPubsMain.asp 44th National Immunization Conference | April 19-22, 2010 | Atlanta, GA