1. Use of Web-Based Reporting Application to Monitor and Improve Data Quality of Immunization Information Systems
Laurel A. Fowler, MPH, CPHQ
CDC Public Health Advisor
Deputy Director
Los Angeles County Immunization Program

2. Overview Registry summary What is the Provider Status Report (PSR)
How registry staff uses the PSR to conduct QA evaluations
Impact of QA follow-up activities
The presentation will cover these topics. Also provide a hard copy of the slide presentation and description of each report contained in the PSR.

3. Los Angeles-Orange Immunization Network (LINK)
Member of CA Immunization Registry (CAIR)
Regional Population: 14 million
Registry Summary:
462,000 records for children < 6 years old w/ at least 2 immunizations
7,255,000 million shot records for children < 6
800+ providers (public health centers, community clinics, hospitals, private providers, WIC agencies, school based clinics, schools, foster care)
The Los Angeles-Orange Immunization Network (LINK) is part of the CA Immunization Registry (CAIR), which is in the process of being consolidated into a statewide registry from the currently separate 9 regional registries. Within the LINK region, there are over 14 million residents. LINK currently has a total of 462,000 records for children under 6 with at least 2 IZs recorded (approx 40% of children <6) and a total of 7,255,000 shot records for this age group. LINK has a total of 1.3 million patient records and 19.2 million shot records. We have over 800 providers using LINK including public health centers, community clinics, hospitals, private providers, WIC agencies, school based clinics, schools and foster care. (Data as of 3/1/09)

4. Why is a QA tool needed?
Measure providers’ use of the registry over time
Measure progress using concrete and measurable data quality indicators
Identify areas needing improvement, provide training and/or additional support
Give providers feedback and opportunities for them to give us feedback
We felt there was a need to develop a tool to help us evaluate the data quality of immunization records entered by providers based on standard criteria. Not only for us to give providers feedback on their use, provide recommendations, and identify areas where additional training and support might be useful, the PSR also gives providers an opportunity to give us feedback on the software, our services, and explain why they think they are having trouble and how they feel we can help them.

5. Overview of PSR
Application is built in ASP.NET, using Visual Basic language and SQL database
Intranet web application allows secure internal staff access only
Independent from registry software
5 major report categories, 33 unique queries
The PSR application was originally built in 2005 and has been expanded and improved regularly over the years. It is in ASP.NET platform. Queries are programmed in visual basic language using a SQL database to store results. The application is separate from the registry software but it queries the live registry database, so it provides real-time data. It is an intranet application and only specified Registry staff have access to it. Queries are grouped into 5 major categories, with at total of 33 unique queries that can be run.

6. Major Indicators Evaluated
Provider Reports: login activity, immunization data, inventory activity, reminder recall, duplicate records
Patient Reports: patient records, patient status, incomplete patient records
Refusal Reports
WIC Reports: read/create record, total transcribed shots, patients claimed, record accessed/read
Other Reports: adverse reaction, waivers, notes, 90 days no shot, provider list
Currently, the 5 categories of reports are: Provider Reports, Patient Reports, Refusal Reports, WIC Reports and Other Reports.

7. This is the main page of the PSR application website, with description of what the PSR does. It is password protected.

8. Reports searchable by Provider, Start/End Dates
Report description and specifications
The PSR has a menu of all categories of reports on the left. Reports are searchable by provider and any start/end dates if applicable.
Reports are categorized into Provider Reports, Patient Reports, Refusal Reports, WIC Reports or Other Reports

9. Patient names can be shown or hidden by checking box
Provider A
Records can be sorted by any underlined column headings (ID, Last Name, First Name, Gender, etc.)
Reports are also exportable to Excel for easier downloading, saving and ing to providers. Reports can also display or hide patient names by checking or un-checking the box “show patient name”. Report can also sort by any underlined column headings for easy viewing preferences.
Patient names can be shown or hidden by checking box
Reports are easily exportable to excel for downloading

10. Incomplete Records Reports
Identifies records that have:
Empty Immunization Histories
Empty Address Fields
Empty Phone Number Fields
Gender listed as ‘unknown’
VFC Status listed as ‘unknown’
Empty Body Site Fields
One of the major areas we identify issues is incomplete records. This category of report identifies missing and incomplete information in important fields. This report allows providers to review records with incomplete information and update them. (All are optional fields in the software except two: Gender and VFC status. For Gender, a gender must be selected from a dropdown menu, but the report report identifies where Gender has been selected as ‘unknown’. Similarly, for VFC, a status must be selected, but this report identifies where VFC status has been selected as ‘unknown’).

11. Inventory Activity Reports
Displays provider inventory activity:
Number of vaccinations given and deleted
Number of inventory adjustments made
Number of transcribed historical records
Negative or zero volume vaccine lots
Duplicate vaccine lots
The registry software has a vaccine inventory module so that real-time entry of vaccines administered should provide an accurate accounting of vaccine on hand. The inventory reports in the PSR help Registry staff identify inventory problems the provider is having. Maintaining an accurate inventory helps providers better track their vaccines and produce more accurate reports – e.g., for VFC.

12. WIC Agency B
Special queries for WIC agencies to show number of patients created by WIC that are claimed by regular providers
The PSR includes additional reports for WIC agencies, to help them track information they need for program and grant requirements. We have WIC agencies that are read-only and also some that enter historical IZs. The Patient Read Report displayed here identifies how many times specific staff have accessed a record in the registry. This is used as a proxy to count encounters for assessment and referral activity.

13. Data from all reports are summarized in a word document
Provider A
Data from all reports are summarized in a word document
Important indicators are highlighted
The data from all of the individual queries are placed in a Word document. We highlight primary issues of concern. We also provide the specific reports to the provider with the detailed information as needed.

14. 2nd page of report gives recommendations on improving specific data quality issues.
This is the second page of the PSR Summary Report. Specific recommendations are provided to help providers understand what steps they can take to improve data quality, based on the results found in the PSR queries.

15. How PSR is Used in Practice
Providers receive reports at 1-month, 3-month, 1-year post-implementation and then annually
On-site visits are conducted for the 1-month report and if subsequent reports show continuing or new problems
Providers are given recommendations to improve data quality based on their results
Additional training and support given as needed
All providers receive reports at a minimum of 1-month, 3-months, 1-year post-implementation and then annually thereafter. A site visit is conducted at a minimum for their 1st report. Additional reports and additional site visits may also be conducted if problems persist or new problems arise. Sometimes reports are reviewed over the phone. If a provider is doing well, future reports are mailed or ed to the provider. We have a staff member dedicated to working with provider sites that are struggling with their use. This staff member works with the provider to identify ways the problems can be resolved and provides or arranges additional support including additional training, helping the provider revise their workflow, providing record entry support and/or computer equipment loans.

16. Provider Progress To Date
Based on PSR results for 194 clinical providers who have used the registry for 2 or more years:
72% showed improvement from 3-month to 1 year reports (140/194)
69% showed improvement from 1 year to 2 year reports (133/194)
51% showed continuous improvement over 2 years (98/194)
As of 12/08, there are 194 clinical providers who have been using the registry for 2 years or more. Clinical providers are those who give immunizations. Of these 194 providers, 72% showed they have improved in the major categories of indicators from 3-month to 1 year reports, 69% of these also showed continued improvement from 1 year to 2 year reports. 51% improved at both the 3-month to 1 year and 1 year to 2 year mark, showing consistent improvement over period of 2 years.

17. How Improvement was Measured
Six indicators were assessed:
% records with empty immunization history
% records with change in patient status
% records with missing address
% records with missing phone number
% records with missing admin by/body site
# of inventory lots showing as negative/zero volume, duplicate or expired
Improvement was defined as at least 3 or more indicators showing improved percentages from the previous report
For this activity, we looked at 6 specific indicators. Improvement was considered if the provider showed improvement on at least 3 of the 6 indicators from report to report.

18. Benefits of the PSR and Follow Up
Allows registry staff to identify any problems or training needs that may not have been evident to the provider or LINK otherwise
Allows providers to see progress in their use of the registry
Helps registry staff prioritize and target follow up efforts with providers having the most problems using the registry or those least active after deployment
We have found the PSR key in identifying problems providers are having. Providers report that they also feel the PSR is very useful. In both cases, the PSR process often identifies problems that neither LINK nor the provider would have known about otherwise. It is important to note that the PSR is also helpful not only to identify problems, but to provide reinforcement and encouragement to providers who are doing well. The PSR has been key in helping LINK prioritize and target follow-up to those providers who need assistance the most.

19. Future Plans
Add additional reports to the PSR that would be helpful to registry, immunization program and provider staff
Provide PSRs to immunization program and VFC staff to incorporate results into QAR and AFIX activities
Possibly being integrated into the CA Immunization Registry for statewide use
We are currently identifying additional queries and reports to add to the PSR that would be helpful to Registry, Immunization Program and provider staff. We have also recently implemented a more formal process for making the PSRs available to immunization program staff involved with QAR and AFIX activities. And we are working with the State Registry office to have the PSR integrated or made available to Registry staff in other regions throughout the State.

20. For More Information Contact: Yang (Lucy) Yang, MPH
LINK Quality Assurance Specialist
Please feel free to contact Lucy Yang at LINK if you have any questions or would like additional information on the PSR or LINKs QA process. Thank you.