Analysis of VFC Provider Quality Assurance Survey Data Pamela Mathison, MA BSN RN Lori Rizzo, MBA BSN RN Texas Medical Foundation Vaccines for Children Project National Immunization Conference March 22, 2005

In Collaboration with: Texas Department of State Health Services Jack Sims Director, Immunization Branch Karen Hess Vaccine Services Group

Objectives 2002-2004: Identify Trends Compare site survey data over three years 2004: Search for Significant Indicators Identify indicators associated with CASA rates 2005+: Forecast the Future Using data results in VFC program planning

TMF Overview Private Non-profit Organization (1971) Physician Directed Quality Improvement/Quality Assurance/Peer Review Contracts Centers for Medicare & Medicaid Services – focus on Medicare population State health departments for VFC quality assurance/AFIX site visits Other organizations and agencies

DSHS-TMF Partnership Initial contract for TVFC QA site visits in fall of 2001 Used paper-based TVFC site survey tool developed by DSHS Developed a set of guidelines and instructions for performing VFC site visits and completing the TVFC survey tool & CASA report 354 TVFC visits completed at private physician offices October-December – majority are a combined VFC/AFIX visit

DSHS-TMF Partnership Renewal of contract for 2002 TMF develops a TVFC electronic site survey tool based on previous paper tool and refinement & addition of items for review Guidelines and instructions are further refined and created for completing the new tool 831 VFC site visits completed at a variety of types of providers – majority are a combined VFC/AFIX site visit

DSHS-TMF Partnership Renewal of contract in 2003 Refinement of electronic tool and instructions 2,408 VFC site visits conducted – majority were a combined VFC/AFIX site visit Renewal of contract in 2004 2,405 VFC site visits conducted – majority were a combined VFC/AFIX site visit

Components of a TVFC QA Site Visit The TMF reviewer performs: VFC site survey CASA (if children in age range) Immunization Record Review Tool Vaccine inventory accountability calculations: “gain/loss” & stock levels Education, feedback and materials Re-enrollment

Electronic Data Collection The electronic tool is designed for: Easy data input, review and modification Accurate & reliable data collection Electronic data storage & abstraction TMF Registered Nurse reviewers have: Comprehensive VFC-related training Clear-cut detailed instructions Continued guidance & support Site survey reports undergo: A two-tiered quality assurance review Final verification by DSHS staff

Objective I: Identify Trends 2002-2004 Retrospective Study: 562 VFC providers Identification & criteria: Site visits in 2002, 2003 & 2004 Private providers, public & private hospitals and schools, FQHC/RHCs Site visits performed by TMF reviewers

Objective I: Identify Trends 2002-2004 Data Collection Direct observation, interview, document & medical record review Data Verification & Validation Tool design limited data entry error Post-survey quality assurance performed Validation with data abstraction

Objective I: Identify Trends 2002-2004 VFC Survey Data 2002: 35 items 2003 & 2004: 40 items CASA rates: 4:3:1

Objective I: Identify Trends 2002-2004 % Compliance Improvement on VFC Survey 35/40 33/40 28/35

Objective I: Identify Trends 2002-2004 “Most Improved” Indicators Bottles of water: 48% (25→73%) Plug guards: 39% (38→77%) Parents informed of state registry: 29% (51→80%) Vaccine storage & handling chart: 28% (61→89%) Current VISs: 26% (46→72%)

Objective I: Identify Trends 2002-2004 “Backsliding” Indicators Freezer temperatures: -6% (93→87%) Refrigerator temperatures: - 6% (96→90%) Monthly vaccine inventory: -5% (98→93%) Harmonized vaccine schedule: -3% (76→73%) All TVFC vaccines routinely stocked -2% (92→90%)

Objective I: Identify Trends 2002-2004 Immunization Practice Areas Vaccine Storage Basic Measures Decreasing Barriers Vaccine Storage Proactive Strategies Minimizing Missed Opportunities Vaccine Inventory Management Record-keeping (& Registry) VAERS

Objective I: Identify Trends 2002-2004 % Improvement in Immunization Practice Areas 2002-2004

Objective I: Identify Trends 2002-2004 % Improvement in Immunization Practice Areas 2002-2004

Objective I: Identify Trends 2002-2004 Indicators with ≥ 98% current compliance 1. Ice packs in the freezer 2. VAERS: Possible adverse events explained 3. VAERS: Information provided on reporting adverse events 4. Vaccine provided regardless of ability to pay 5. TVFC vaccines separated from private stock

Objective I: Identify Trends 2002-2004 Indicators with ≥ 98% current compliance 6. TVFC eligibility screening performed 7. Appropriate refrigerators and/or freezers 8. Vaccines simultaneously administered 9. Vaccine stock rotated 10. Underinsured children vaccinated with TVFC stock

Objective I: Identify Trends 2002-2004 % of providers with changes in CASA 4:3:1 rates 2002-2003 2003-2004 % providers with:

Objective I: Identify Trends 2002-2004 Median CASA 4:3:1 rate improvement! 2004 4:3:1 rate = 77% 2003 4:3:1 rate = 70% 2002 4:3:1 rate = 58%

Objective II: Significant Indicators 2004 Wouldn’t it be great if we could predict what indicators contributed most to achieving a high CASA rate?

Objective II: Significant Indicators 2004 Which indicators are associated with CASA rate? “TOP 3” P<0.0001 R2=20.7 Complete Immunization Histories Recall Systems Immunization Documentation

Future Focus on Key Indicators Influence on CASA (rank) % Compliance Change 2002-04 Current Compliance Immunization Histories 1 + 14.3% (9th) 90.3% Recall System 2 + 7.4% (13th) 78.0% Immunization Documentation 3 + 13.0% (11th) 60.5%

Limitations Convenience sample Quality improvement process adds variability Changes in provider practices affect results External influences add “noise” Subjective data less reliable Nominal “yes/no” scale limits analysis Site visit feedback & education individualized Past does not necessarily predict the future!

Final Thoughts “The problem with measurement is that it can be a loaded gun – dangerous if misused and at least threatening if pointed in the wrong direction.” Dennis O’Leary, President JCAHO

Final Thoughts Use quantitative analysis as an adjunct to qualitative review, communication and insight… …with a focus on continuous quality improvement

For more information, contact: Pam Mathison, VFC Project Manager pamm@tmf.org Lori Rizzo, VFC Asst. Project Manager lorir@tmf.org Texas Medical Foundation Austin, TX 1-800-725-9216 www.tmf.org