1. Evaluating IIS Best Practice Operational Guidelines: Emerging Trends and Challenges
Warren Williams, MPH
Lead Public Health Analyst, CDC,
National Center for Immunization and Respiratory Diseases
The 44th National Immunization Conference
Hyatt Regency Hotel, Atlanta, Georgia
Tuesday, April 20, 2010, 4:00 PM
International Ballroom North
National Center for Immunization and Respiratory Diseases
Immunization Information Systems Support Branch

2. and do not necessarily represent the views of the funding agency
Presentation collaborators
Elaine Lowery, JD, MSPH - Senior Public Health Advisor, Public Health Informatics Institute
David Lyalin, PhD - Consultant, Northrop Grumman - CDC Information Technology Support Contract
Nichole Lambrecht, MSc - Project Manager, Kansas Immunization Registry – KSWebIZ
Sherry Riddick, RN, MPH - Immunization Registry Operations Manager, CHILD Profile
Cynthia Sutliff - Executive Director, American Immunization Registry Association (AIRA)
Vikki Papadouka, PhD, MPH - Director of Research and Evaluation, NYC Department of Health and Mental Hygiene
First a special thanks to the collaborators on this presentation.
The findings and conclusions in this report are those of the author(s)
and do not necessarily represent the views of the funding agency

3. MIROW
The Modeling of Immunization Registry Operations Workgroup (MIROW) of the American Immunization Registry Association (AIRA) develops Best Practices for IIS functionality.
MIROW Goals
Promote operational consistency
Foster communication and collaboration
Increase credibility of IIS
MIROW Approach
Facilitated collaboration of contributing IIS experts to evaluate and combine various operational practices and achieve a consensus regarding best practices.
Use of business analysis and modeling techniques to support development of best practices .
Business modeling is a discipline of systems engineering that facilitates analysis and promotes insight of the business’s functional and operational aspects independent of technical implementations. Business modeling involves the use of structured techniques to capture enterprises’ process and organizational components, as well as rules and regulations.
Systematic analysis of public health operations with formal business modeling techniques can promote systems thinking and help to scrutinize and improve public health processes.

4. Why is consistency of IIS operations important?
Inconsistency among IIS negatively affects overall data quality and usefulness of registry information.
MIROW guidelines are intended to support uniform alignment of IIS operations.

5. How MIROW works MIROW efforts are guided by the Steering Committee
Warren Williams, Co-Chair,
Elaine Lowery, Co-Chair,
Operational topics for analysis are selected based on feedback from IIS community (assessments, surveys, ad-hoc meetings at NIC)
Panel of experts is assembled for each topic
A combination of face-to-face analysis sessions and web-based teleconferences
External reviews of developed guidelines
Assessments and surveys on implementation in IIS

6. MIROW participants Subject matter experts Steering Committee
Facilitation Teams
AIRA staff
State IIS
External Reviewers
Technical Editors at CDC 6

7. Guidelines developed
: Reminder/Recall in Immunization Information Systems
2007: Data Quality Assurance in Immunization Information Systems: Incoming Data
2006: Vaccination Level Deduplication in Immunization Information Systems
2005: Management of Moved or Gone Elsewhere (MOGE) Status and other Patient Designations in Immunization Information Systems
: (Pilot) IIS-VAERS Collaboration for Vaccine Adverse Events Reporting
Recommendations documents are available for download
at the AIRA web site:

8. Topics Guideline document released Face-to-face meeting Experts’ panel
size
Guideline document
highlights
Reminder/Recall in IIS
April
2009
October 2008
2.5 days
Tampa, FL 13
29 Principles
23 Business rules
30 General Recommendations
Data Quality Assurance in IIS: Incoming Data
February 2008
August 2007
Atlanta, GA 11
13 Principles
32 Business rules
Vaccination
Level Deduplication in IIS
December
May 2006 20
9 Principles,
20 Business rules,
23 Illustrative scenarios (examples)
Management of
Moved or Gone Elsewhere (MOGE) Status and other Patient Designations in IIS
December 2005
August 2005 16
6 Statuses defined on the Provider level,
5 Statuses on the Geographic Jurisdiction level
IIS-VAERS Guide
(pilot project)
2005
June 2004
1.5 days 21
10 Functional standards,
8 Business rules,
11 Alternative scenarios (process)
Topics

9. What do the MIROW recommendations offer?
Peer-reviewed practical guidelines
Experts from 24 IIS participated in MIROW sessions since 2005, as well as many IIS partners and stakeholders from public health programs, federal agencies, regional and county health departments, IT vendors, consultancies, and private registries.
Guidelines include:
A set of agreed upon terms and definitions
General principles, specific business rules and recommendations to apply
Illustrative examples and templates
Strategies on how to address problems, issues, and barriers

10. Promoting the MIROW guidelines
Presentations, workshops, and organized ad-hoc meetings at annual national immunization conferences
Brochure-style mini-guides summarizing each of the guideline documents - to encourage IIS staff to learn more about the best practices recommendations
Self-assessment tool for the data quality assurance guidelines - for gap identification between current IIS practices and the MIROW best practice recommendations (has been presented at an AIRA webinar)

11. Evaluation approaches
After every workshop, a questionnaire was completed by participating experts to gather feedback on various aspects of the best practice development efforts.
Questions regarding utilization of MIROW guidelines were included in the Immunization Information Systems Annual Report (IISAR).
A web-enabled survey was conducted to examine awareness and use of the patient status guidelines.
Additionally, informal feedback gathering approaches were utilized, e.g., case studies of guidelines implementation in state IIS, ad-hoc meetings at immunization conferences, and external reviews of guidelines.

12. Evaluation results summary
Use of MIROW guidelines by IIS according to IISAR data:
- 46% in % in 2008
Web survey results - response rate: 41% (30/74) -18/19 participants familiar with guidelines found them to be at least somewhat useful
Average satisfaction among MIROW contributors: / 4

13. Expert’s surveys: conducted after face-to-face sessions
Scale 1 – 4: 4 – highest/most positive,
1 – lowest/least favorable

14. Results: Implementation
13 out of 19 of the respondents who were familiar with the Guidelines have implemented or attempted to implement the Guidelines
7 out of 13 of the respondents who had attempted to implement the Guidelines encountered one or more barriers to implementation
Implementation Barriers:
BR inconsistent with other programs and/or business requirements
Lack of staff time to reconcile competing business requirements
Competing funding priorities
Considering implementation in application upgrade
Response rate = 41% (30/74)

15. MIROW activities being carried out by IIS Sentinel sites
Project Sites
MIROW Activities AZ MI ND
NYC OR WI
Data Quality Improvement X
Vaccine De-duplication
Moved Or Gone Elsewhere (MOGE)
these efforts are funded with ARRA resources 15

16. Implementations in KS and WA
The Kansas IIS (KSWebIZ) team developed new reports at both the provider and system levels to enhance overall data quality measures in accordance with the MIROW best practice guidelines on data quality.
Using a team/workgroup approach, the Washington IIS reviewed the MIROW data quality guidelines to compare current data loading and quality check practices with the best practices guidelines. As a result, 21 of the 32 MIROW data quality business rules were adopted and then current IIS business processes, policies and procedures were revised and improved to support the adopted business rules.
Implementation in the Kansas Immunization Registry (KSWebIZ). The Kansas IIS (KSWebIZ) team developed new reports at both the provider and system levels to enhance overall data quality measures in accordance with the MIROW best practice guidelines on data quality. First, the team conducted a gap analysis to identify where existing data quality procedures in KSWebIZ differed from the best practices guidelines. Next, those gaps that had to do with assurance of data timeliness, completeness, and accuracy were evaluated and prioritized. Finally, reports were developed that gave the IIS and healthcare providers the details needed to address the gaps.
 Implementation in Washington State IIS (CHILD Profile). The Washington IIS (CHILD Profile) began reviewing established methods for data loading and quality checks. The intent of the review was to ensure that the amount of data being submitted to the IIS from healthcare providers and other organizations continues to increase and that the quality, timeliness and accuracy of that data continue to improve. Using a team/workgroup approach, the Washington IIS reviewed the MIROW data quality guidelines to compare current data loading and quality check practices with the best practices guidelines. As a result, 21 of the 32 MIROW data quality business rules were adopted and then current IIS business processes, policies and procedures were revised and improved to support the adopted business rules. These changes led to improved quality of data in several key areas.

17. AIRA regional forums
Upcoming in CO: Half day to focus on the MIROW Data Quality and the MOGE/Patient Status Best Practice Guidelines
The Data Quality Best Practices guidelines were discussed at the Midwest and the Southeast Regional Forums
MIROW mini-guides on all Best Practice Guidelines are distributed at all Regional Forums 17

18. Conclusions
Facilitated collaboration of contributing IIS experts resulted in development of common consensus-based guidelines. That have lead to support of immunization program activities with better IIS information.
Implementation barriers related to limited resources and competing priorities occur locally and have to be addressed
While IISAR results indicate awareness and recognition of the MIROW best practice operational guidelines in the IIS community, additional studies are needed to evaluate specifics of guidelines’ acceptance and utilization.

19. Next MIROW topic: IIS-VFC/AFIX Collaboration
A primary focus is resolving and tracking VFC eligibility status and history
Face-to-face meeting scheduled for June 15-17, 2010 in Atlanta, GA
16 experts from state IIS and VFC programs, federal VFC program, vendors, and consultants will participate
Expected guideline release – end of 2010
A primary focus for the new MIROW IIS-VFC/AFIX topic is on resolving and tracking the VFC eligibility status and history, such as:
Patient VFC eligibility – based on the insurance coverage, ethnicity, and state policies (e.g., Universal state)
Vaccine VFC eligibility – based on a funding source, ACIP recommendations and state policies (e.g., VFC Select state)
Provider VFC eligibility – based on the provider’s VFC enrollment status and facility’s VFC eligibility for a certain vaccinations
Vaccination Event VFC eligibility – based on compliance with ACIP recommended schedule and other factors
Additionally, issues of data quality will be one of the major focuses, e.g., determine if and when IIS data is complete enough or correct enough for use in a VFC program. A secondary focus, as time permits, is going to be on other VFC/AFIX programs areas/ activities and data elements needed to support them.

20. Questions?
Copies of the MIROW recommendations documents can be found at the AIRA web site:
Contact Information:
Warren Williams
National Center for Immunization and Respiratory Diseases
Immunization Information Systems Support Branch