1. Analysis of VFC Provider Quality Assurance Survey Data
Pamela Mathison, MA BSN RN
Lori Rizzo, MBA BSN RN
Texas Medical Foundation
Vaccines for Children Project
National Immunization Conference
March 22, 2005

2. In Collaboration with:
Texas Department of State Health Services
Jack Sims
Director, Immunization Branch
Karen Hess
Vaccine Services Group

3. Objectives 2002-2004: Identify Trends
Compare site survey data over three years
2004: Search for Significant Indicators
Identify indicators associated with CASA rates
2005+: Forecast the Future
Using data results in VFC program planning

4. TMF Overview Private Non-profit Organization (1971) Physician Directed
Quality Improvement/Quality Assurance/Peer Review Contracts
Centers for Medicare & Medicaid Services – focus on Medicare population
State health departments for VFC quality assurance/AFIX site visits
Other organizations and agencies

5. DSHS-TMF Partnership
Initial contract for TVFC QA site visits in fall of 2001
Used paper-based TVFC site survey tool developed by DSHS
Developed a set of guidelines and instructions for performing VFC site visits and completing the TVFC survey tool & CASA report
354 TVFC visits completed at private physician offices October-December – majority are a combined VFC/AFIX visit

6. DSHS-TMF Partnership Renewal of contract for 2002
TMF develops a TVFC electronic site survey tool based on previous paper tool and refinement & addition of items for review
Guidelines and instructions are further refined and created for completing the new tool
831 VFC site visits completed at a variety of types of providers – majority are a combined VFC/AFIX site visit

7. DSHS-TMF Partnership Renewal of contract in 2003
Refinement of electronic tool and instructions
2,408 VFC site visits conducted – majority were a combined VFC/AFIX site visit
Renewal of contract in 2004
2,405 VFC site visits conducted – majority were a combined VFC/AFIX site visit

8. Components of a TVFC QA Site Visit
The TMF reviewer performs:
VFC site survey
CASA (if children in age range)
Immunization Record Review Tool
Vaccine inventory accountability calculations: “gain/loss” & stock levels
Education, feedback and materials
Re-enrollment

9. Electronic Data Collection
The electronic tool is designed for:
Easy data input, review and modification
Accurate & reliable data collection
Electronic data storage & abstraction
TMF Registered Nurse reviewers have:
Comprehensive VFC-related training
Clear-cut detailed instructions
Continued guidance & support
Site survey reports undergo:
A two-tiered quality assurance review
Final verification by DSHS staff

10. Objective I: Identify Trends 2002-2004
Retrospective Study:
562 VFC providers
Identification & criteria:
Site visits in 2002, 2003 & 2004
Private providers, public & private hospitals and schools, FQHC/RHCs
Site visits performed by TMF reviewers

11. Objective I: Identify Trends 2002-2004
Data Collection
Direct observation, interview,
document & medical record review
Data Verification & Validation
Tool design limited data entry error
Post-survey quality assurance performed
Validation with data abstraction

12. Objective I: Identify Trends 2002-2004
VFC Survey Data
2002: 35 items
2003 & 2004: 40 items
CASA rates: 4:3:1

13. Objective I: Identify Trends 2002-2004
% Compliance Improvement on VFC Survey
35/40
33/40
28/35

14. Objective I: Identify Trends 2002-2004
“Most Improved” Indicators
Bottles of water: 48% (25→73%)
Plug guards: 39% (38→77%)
Parents informed of state registry: 29% (51→80%)
Vaccine storage & handling
chart: 28% (61→89%)
Current VISs: 26% (46→72%)

15. Objective I: Identify Trends 2002-2004
“Backsliding” Indicators
Freezer temperatures:
-6% (93→87%)
Refrigerator temperatures:
- 6% (96→90%)
Monthly vaccine inventory:
-5% (98→93%)
Harmonized vaccine schedule:
-3% (76→73%)
All TVFC vaccines routinely stocked
-2% (92→90%)

16. Objective I: Identify Trends 2002-2004
Immunization Practice Areas
Vaccine Storage
Basic Measures
Decreasing Barriers
Vaccine Storage
Proactive Strategies
Minimizing Missed
Opportunities
Vaccine Inventory
Management
Record-keeping
(& Registry)
VAERS

17. Objective I: Identify Trends 2002-2004
% Improvement in Immunization Practice Areas

18. Objective I: Identify Trends 2002-2004
% Improvement in Immunization Practice Areas

19. Objective I: Identify Trends 2002-2004
Indicators with ≥ 98% current compliance
1. Ice packs in the freezer
2. VAERS: Possible adverse events explained
3. VAERS: Information provided on reporting adverse events
4. Vaccine provided regardless of ability to pay
5. TVFC vaccines separated from private stock

20. Objective I: Identify Trends 2002-2004
Indicators with ≥ 98% current compliance
6. TVFC eligibility screening performed
7. Appropriate refrigerators and/or freezers
8. Vaccines simultaneously administered
9. Vaccine stock rotated
10. Underinsured children vaccinated with TVFC stock

21. Objective I: Identify Trends 2002-2004
% of providers with changes in CASA 4:3:1 rates
% providers with:

22. Objective I: Identify Trends 2002-2004
Median CASA 4:3:1 rate improvement!
2004
4:3:1 rate = 77%
2003
4:3:1 rate = 70%
2002
4:3:1 rate = 58%

23. Objective II: Significant Indicators 2004
Wouldn’t it be great if we could predict what indicators contributed most to achieving a high CASA rate?

24. Objective II: Significant Indicators 2004
Which indicators are associated with
CASA rate?
“TOP 3”
P<0.0001
R2=20.7
Complete Immunization Histories
Recall Systems
Immunization Documentation

25. Future Focus on Key Indicators
Influence
on CASA
(rank)
% Compliance
Change
Current
Compliance
Immunization Histories 1
+ 14.3% (9th)
90.3%
Recall System 2
+ 7.4% (13th)
78.0%
Immunization
Documentation 3
+ 13.0% (11th)
60.5%

26. Limitations Convenience sample
Quality improvement process adds variability
Changes in provider practices affect results
External influences add “noise”
Subjective data less reliable
Nominal “yes/no” scale limits analysis
Site visit feedback & education individualized
Past does not necessarily predict the future!

27. Final Thoughts
“The problem with measurement is that it can be a loaded gun – dangerous if misused and at least threatening if pointed in the wrong direction.” Dennis O’Leary, President JCAHO

28. Final Thoughts
Use quantitative analysis as an adjunct to qualitative review, communication and insight… …with a focus on continuous quality improvement

29. For more information, contact:
Pam Mathison, VFC Project Manager
Lori Rizzo, VFC Asst. Project Manager
Texas Medical Foundation
Austin, TX