1. Immunization Practices Improvement Initiative (IPII)
A Process Evaluation of a Level 1 AFIX Program
Susan Ashkar, MA
Melissa Ely Moore, MPH
Immunization Program
Los Angeles County Department of Health Services

2. AFIX Program Program Operations Assessment Feedback Incentives
eXchange of Information
Program Evaluation
The elements of an AFIX program are….

3. Assessment and Feedback Activities Los Angeles County Immunization Program (LACIP)
Community Workers
CASA
Nurse consultant
Quality Assurance Review (QAR)
Feedback
Action Plan
Disjointed efforts
Prior to implementing our own AFIX program at the LA County IZ Program already engaged in Assessment and Feedback activities for non-profit health care providers who receive publicly purchased VFC vaccine.
First community workers conduct a CASA. CASAs are done annually.
Within 2 weeks of the CASA, a nurse consultant conducts a Quality Assurance Review (QAR). The nurse spends about 4 hours evaluating the clinic based upon the Standards of Pediatric Immunization Practice and reviewing patient charts. At the end of the observation, the nurse would provide feedback to the provider. Together they develop an Action Plan to correct any deficiencies found. This is signed by that clinic staff member. There were 155 QARs performed in 2004 and 86 CASAs were completed in 2004.
This tended to be disjointed and efforts were not coordinated. Staff may have been working with the same provider, but not working together so we wanted to have more collaboration among disciplines with more of a case management approach. So the Immunization Practices Improvement Initiative was implemented.

4. Immunization Practices Improvement Initiative (IPII)
The goal of the IPII is to raise immunization rates and improve provider practices by:
implementing evidence-based interventions to improve provider immunization practices
instituting routine and proactive follow-up
increasing efficiencies by assessing office systems
increasing coordination and collaboration among LACIP staff
LACIP began an AFIX program we titled the Immunization Practices Improvement Initiative (IPII).
The goal of IPII is to raise immunization rates by:
Utilizing descriptions from the clinics
Discussing everything going on with the clinic

5. Program Operations Level 1 Standards Partially Met
Written strategic plan for AFIX activities
Clearly defined methods for evaluating progress
QAR and CASA results
Method for evaluating the progress at achieving goals
Monthly Case Management Team meeting where goals & timeline for improvement are set
In many ways IPII follows the AFIX Standards that CDC set forth and Nancy just spoke about. I am going to discuss how IPII compares to the AFIX Standards and then look at what we can do to improve.
For Program Operations, we partially meet Level 1 AFIX Standards. We have a protocol which spells out what our AFIX project is to accomplish.
Through the QAR and CASA, we have methods for evaluating our progress.
We also have a method to evaluate our progress at meeting our goals and that is through monthly case management team meetings where we discuss clinics, how to help them improve, and set goals and timelines for clinic improvement.

6. Assessment Levels 1-2 Standards Met
Written CASA and QAR protocols
CASA protocol:
Age range to be assessed months
Inclusion Criteria
Series to be assessed 4:3:1:3:3
Demographics to be collected
MOGE is defined
QAR observation and chart review
As for assessment standards, IPII met Levels 1 & 2. In IPII there is a written CASA and QAR protocol. The CASA protocol lists items such as, the age range to be assessed, the inclusion criteria, series to be assessed which is 4:3:1:3:3 defines what MOGE. The QAR consists of observation – assessing clinics flow, vaccine storage and handling, vaccine administration, and chart review.

7. Case Management Team
Composed of nurses, community workers, research analysts & program managers
Discuss assessments, office practices
Develop improvement plan
What interventions are needed
Who will make the follow-up visits
Frequency of follow-ups
Incentives to offer
Next the clinic is discussed at the Case Management Team Meeting. The team is made up of the nurses, CW, RAs and program managers of LACIP. We are from varying disciplines and educational levels and so bring our different experiences to the discussion. We look at the results of the assessments and discuss overall office practices. Team members work together to formulate the best plan for each office so that each office may benefit from the resources LACIP has to offer. We decide what sorts of interventions are needed based upon the deficiencies found in the office, who will conduct the follow-up visits, how frequently follow-up will be needed, and also if any incentives can be offered for doing well or improving.

8. Case Managers

9. East Area Field Unit Susana Sevilla, Case Manager Karen Roby
Judy Graham
Lisa Velasco
Vickie Washington
Mary Porras
Anthony Williams
(not pictured)
Kay Park (not pictured)
Sonia Martinez
Sherry Russell
Susana Sevilla, Case Manager

10. East Area Field Unit

11. Southwest Area Field Unit
Claudia Davila
Cruz Porras
Arnold Hartoonian
Na’ama Hadar
Paul Miller
Ricardo Ibarra
Jose Salazar
Anna Salinas (not pictured)
Shawntay Davis
(not pictured)
Andrew Pourmohsen
Lucinda Clare
Pierre Nsilu, Case Manager

12. Southwest Area Field Unit

13. North Area Field Unit Susana Sevilla, Case Manager Lidoosh Hartoonian
Becky Bravo
Patricia Klee
Eva Rodriguez
Sevak Karapetyan
Bethany Lorca
(not pictured)
Susana Sevilla, Case Manager

14. North Area Field Unit

15. Feedback Levels 1 –2 Standards Met
Plan is presented to provider
Assessment results reviewed
Goals set, Healthy People 2010 discussed
LACIP interventions
In-services, training offered
Reminder/Recall, documentation, immunization update, simultaneous administration, missed opportunities
Follow-up visits/phone calls to ensure change implemented
The feedback session would follow the assessments after the CM Team has had an opportunity to discuss the clinic and make its recommendations. In IPII, feedback also meet Standards for Levels 1 & 2.
A team member would provide feedback to the provider based upon the discussion of the CM team. The clinic’s assessment results would be reviewed, clinic strengths and weaknesses, and the suggested improvement plan reviewed and with input from the clinic, goals set for improvement and the ultimate goal being to reach the Healthy People 2010 immunization goals are discussed. Incentives for meeting those goals discussed, more about incentives in a moment.
The improvement plan includes what we, LACIP, can do for a clinic to assist in meeting these goals. We can schedule on-site trainings and in-services on topics such as reminder and recall, proper documentation, immunization update. We let the clinic know that we can provide on-going follow-up afterwards to answer questions and provide further technical assistance and ensure that the change we are going for has been properly implemented.

16. Incentives Level 1 –2 Standards Met
Written Incentive Distribution Policy
Clearly defined list of formal and informal incentives
certificate of appreciation signed by LACIP Director and Medical Director
gift cards, stethoscopes
promotional items:
stress balls, pens, pocket planners
As for incentives, at LACIP we have a written incentive distribution policy. This defines when formal and informal incentives should be given out and what is appropriate. Informal incentives are things like print materials, literature and formal incentives acknowledge improved or high coverage levels or good practices. To that end we have certificates of appreciation when a clinic has done well, gift cards & stethoscopes and then smaller items such as pens.

17. eXchange of Information Level 1 Standards Met
Print resources available
Vaccine Information Statements (VIS)
current immunization schedule
Inservices offered monthly and as needed
Info on satellite conferences
Access to LACIP website and others
At feedback
The eXchange portion of our AFIX program includes having available print resources such as current VIS and immunization schedules.
Our nurses offer in services on particular topics monthly and also in service their providers as needed if they see a need or if a clinic makes a request.
LACIP also offers information on satellite conferences, our website and other helpful website addresses are made available.
Exchange of information also occurs at the feedback sessions when members of the case management team provide information and training.

18. Program Evaluation Level 1 –2 Standards Met
Utilize Access database to document:
assessment results
site visits
clinic interventions by LACIP staff
staff responsible
date to complete intervention
incentives given
Information is gathered at CM meetings, via telephone and
As for Program Evaluation, we have developed an access database to document our clinic interactions – site visits, interventions by LACIP staff, which staff member is responsible and by what date, were any incentives given and assessment results. We gather this information primarily at our CM Team meetings, but also through telephone and .

19. Program Operations Toward Levels 2-3
Clearly defined method for identifying and recruiting providers for future IPII
Established relationships with Department of Health, Community Clinics (contract & non-contract), VFC
Work with health plans and IPAs
What I have just reviewed is how IPII meets many of the Standards for a Level 1 and sometimes 2 Level AFIX program. Now I will go over some ways that we plan to improve IPII so that it meets even more of the Standards for a higher level AFIX program and becomes a more effective program.
For Program Operations, we can improve upon our ability to identify and recruit providers for our AFIX program.
LACIP has established relationships with many clinics already. We work with community clinics, some we have contracts with to deliver immunization services and others we give vaccine to. We also work closely with the health department clinics and Vaccines for Children providers. With about 5, 000 providers in Los Angeles County, there are many providers who do not fit into any of these categories who we would like to partner with. We think that working with health plans and IPAs are probably good avenues that we would like to explore in the future.

20. Assessment Toward Levels 2-3
Monitoring AFIX staff members implementation of the assessment protocol
Updating QAR protocol
Documentation of interaction with providers by staff
Maintenance for clinics doing well
Our assessments can be brought up to a higher level by monitoring our staff members of the implementation of the protocols available. To that end we are updating our QAR protocol. We also plan to document in greater detail the interaction of staff with clinics.
Also, for those clinics doing well that don’t require interventions and follow up, we are working on a document to follow clinics that are pretty much in maintenance mode. We just need to know someone will check in on a clinic and make sure everything is okay between assessments.

21. Feedback Toward Levels 2-3
Standardized interventions
Resource materials for provider
binder
Services summary
vaccine usage
assessment history
recommendations
To improve upon our Feedback standards which suggest we explore and pilot innovative methods for engaging providers and provide ongoing assistance, we are developing a set list which contains a range of interventions we can offer our providers when they are deficient in any of the Standards of Pediatric Immunization Practice observed during the QAR. We will use this during our CM meetings to determine which intervention is best suited for the provider and which team member is most suited to carrying out that intervention then what sort of follow-up will be needed. This will enable us to have a more clearly defined process for developing the improvement plan.
To complement that, we are preparing a binder containing resources that will assist with the interventions we have proposed this can assist the clinic to meet the Standard they were deficient in.
Also, we think that a helpful feedback session will include a summary sheet for each provider listing the clinic’s assessment history, their vaccine usage the last few years, goals and a timeline for improvement. This could help the provider see how they are doing and how they have done. It can give them context for not just knowing this year’s rates, but how they’ve done overall.

23. eXchange of Information Toward Levels 2 - 3
Sponsor provider exchange & recognition event Summer 2005
Develop a newsletter highlighting best practices to IPII providers
Grand Rounds
For an improved eXchange of information among providers, we plan to sponsor a provider recognition event this summer. A sort of gathering with food and a guest speaker with the offer of CMEs/CEUs. We want to recognize providers who have done well and also excite other providers to want to strive for more.
We also plan to develop a newsletter where we can highlight clinics and get providers to share what has worked well
In addition, using grand rounds as a forum to discuss good practices

24. IPII First Year 36 Community Clinics (Contract)
year CASA average
76% UTD*
2004 CASA average
88% UTD*
*4:3:1:3:3 series at months of age
I wanted to share with you results we have after our first year of IPII. We began the end of In our first year of implementing IPII, we worked with the clinics that we have contracts with. LACIP has contracts with 21 agencies with a total of 36 clinics.
To give you a general idea, these clinics had a 3 year CASA average of 76% UTD for 4:3:1:3:3 series at months. High!
The CASAs done at the end of our first year 2004 averaged 88%. Even higher!

25. First Year Project Report Community Clinics (Contract)
2003 and 2004 CASA Results
4:3:1:3:3 Series
24-35 months
This is a graph demonstrating the UTD rates of clinics studied the first year of IPII. The red bars show the 2004 UTD rate and the line across shows the 2003 rates. Again these are for 4:3:1:3:3 series at months. Not all of the clinics had CASAs performed in both 2003 and If there were not at least 20 children in the months age range, they were assessed using a Casita which doesn’t give us a rate.
17 of the clinics began using registry during the first year of IPII. Potential influence on UTD rates.

26. Challenges Providers LACIP staff Limit visits
Provide solutions, attainable goals
LACIP staff
Change
Workload
Consistent follow-up
Frequent Case Management meetings
By our UTD rates, it looks pretty successful.
There were many challenges to implementing this AFIX program.
We hear often from providers that they receive a lot of visits and we know that they have many priorities. We are talking about two assessment visits, a feedback visit, and then follow-ups.
In order for AFIX to be relevant, we need to offer providers our assistance, solutions to problems and goals they can attain. We should focus on a few interventions that will truly assist the practice, fit in with their busy office system, incentives. Biggest bang for their buck.
For LACIP staff, it is change. We used to do CASA, QAR and the nurse would on the spot, do the feedback which is great to limit the number of visits, but we don’t get our CM team discussion in that way. Changing this system brings another clinic visit which takes our time and theirs. Our feedback hasn’t yet worked the way it was intended. We still do feedback right after the QAR because of the concern of requiring another visit.
Also, change is not an assault on one’s ability to do their job well. Whereas before our staff operated quite independently and I mentioned our duties may have been disjointed, and we have had to learn to work together more and give up some autonomy, but we think our case management approach will be more successful for our providers.
Workload-Having to provide consistent, standardized follow up is a new way of doing things. Still working on this. This has been a challenge. Documenting this information. We know our staff does a lot of work with their providers and we want to have something to show for it so we can determine what interventions worked, which were not so successful.
Case management meetings began every other week. This became too time consuming and we dropped them to 1X/month. We hope this keeps our feedback timely. Yet to explore.

27. Next Steps Identify medium sized IPA to work with
Fully meet Standards for Levels 2 and 3
Continue to evaluate and improve IPII
With IPII our next steps are to expand our provider base and work with those providers who haven’t had so much interaction with the public sector. We would like to find a medium sized IPA to work with.
We will try to meet those Level 2 and 3 Standards in each aspect of AFIX.
And finally we continue to evaluate and improve IPII since we want it to a useful way to assist our providers.

28. Susan Ashkar
Immunization Program
3530 Wilshire Blvd Suite 700
Los Angeles, CA 90010
Phone #
Fax #