Using a Web-Based Reporting Application to Evaluate an Immunization Registry Software Transition Letty Cherry Kreger, MHA California Immunization Registry (CAIR) Inland Empire Region

Outline Background Evaluation Objectives Preparing for the Evaluation PSR Reports Used in Evaluation Process and Results Barriers Next Steps

Background The Provider Status Report (PSR) was developed by the LA-OC Region of CAIR Web-based application Contains several queries used to assess CAIR use and quality of data It became available to all CAIR regions in 2009 Next logical step – use PSR to evaluate the transition from VaxTrack to CAIR software

Evaluation Objectives How many providers were using the inventory feature? Did the inventory feature make an impact on provider use of the registry? If providers did not use inventory, did they enter any immunizations? If they did not enter any immunizations, how long had it been since their last log-in to system? What are their barriers to using CAIR, if any? How well are the high use providers adjusting to CAIR?

Preparing for the Evaluation Determine which PSR reports to use Inventory Transactions Records without Inventory Transactions Log-in Activity Identify providers to evaluate Run baseline report Compare to historical data Contact providers Monitor and run follow-up report

PSR Reports – Inventory Transactions Identifies the number of shots given and deleted from inventory during the specified time frame

PSR Reports – Records without Inventory Transactions Identifies records containing only transcribed immunization history. Active patients only. May be a sign that provider is not using or struggling with Inventory

PSR Reports – Login Activities Identifies login information for users (days since last login, total logins Is provider at least looking up records? Does this provider have an EMR?

Selecting Providers to Evaluate All “high use” providers under VaxTrack Random Selection Small sample size (n = 34) Checked records in Provider Management Database (PMDB) Offices Closed = 1 Offices Trained = 29 Offices Not Trained = 4

How Many Providers Used CAIR? Evaluation Indicator 07/13/2009 to 12/08/2009 07/13/2009 to 04/12/2010 Sample Size 34 # Providers Trained 29 31 # Using Inventory 18 # Entering Transcribed Immunizations 6 4 The number of providers using Inventory remained consistent The number of providers entering transcribed data decreased by 33% Time Period – July 13 to December 8 56% providers using inventory 74% inventory and/or transcribed immunizations

Findings about CAIR Use The clinics using the inventory function in December continued using the inventory function. Two clinics entering transcribed immunizations stopped entering immunizations One provider lost primary user, other uses an EMR

How Many Providers Logged Into CAIR? Log In Frequency 07/13/2009 to 12/08/2009 07/13/2009 to 04/12/2010 Within Last 7 days 19 21 Within Last 8 – 30 days 4 2 Within last 31-90 days 1 With last 91-180 days 5 > 180 days ago 3 Never Untrained providers (4) not included 23 providers consistently logging into registry

Findings about Log-in Frequency and Use of CAIR All provider sites using Inventory logged into CAIR within the past week (most within the past 24 hours). Sites entering transcribed vaccines logged into the registry within two weeks. Providers who were not entering data into CAIR did not log in to check children’s immunizations. McGinness – transcribed some immunizations shortly after go live, then dropped off (EMR)

Barriers to CAIR Use Excludes providers who had not been trained Inventory Use: 50% = Time 50% = Training CAIR Use: 75% = Time 25% = Staff Turn-Over

Did the Transition Affect the Quantity of Immunizations Entered? Quantity of Data Entry 07/13/2009 to 12/08/2009 07/13/2009 to 04/12/2010 New Immunizations Entered Same Time Period Last Year (2008-2009) 22,014 40,837 Current New Immunizations Entered 12,514 29,653 Percent Change -43% -27%

Possible Explanations to the Change in Volume Learning curve and additional need for training Lack of retention-based follow-up by staff No deployment visits No follow-up site visits Lack of feedback to provider office (reports) Lack of staff time to enter data DeMola – one high use user who entered most of the records was not working as much Staff time – audits, increased patient load

Next Steps Resume/continue with regular retention activities Quarterly phone calls to assess registry use and training needs Offer retraining as needed Deployment visits with any site receiving additional training Timely and regular feedback will help increase registry adoption

Next Steps, continued Follow-up Visits to all transitioned providers Introduce Provider Status Reports to office manager Discuss other CAIR reports and benefits Implement action plan to improve CAIR use Continue to monitor providers’ registry use

Lessons Learned Make sure retention activities fit into the transition plan. Be prepared to give feedback to providers early and often. Teach providers how to measure their own success.

Acknowledgements California Immunization Registry (CAIR) State Level Staff CAIR - Los Angeles-Orange County Region Staff CAIR - Inland Empire Staff

For questions or more information: Letty Cherry Kreger, MHA Regional Manager, CAIR Inland Empire lcherry@co.riverside.ca.us