1. Monitoring & Evaluation for Routine Immunization: Data For Action
Using information to improve routine immunization services

2. Presentation 2 Objectives
Be familiar with the role of information in routine immunization
Be able to use, modify or develop tools for monitoring or improving data quality and use at health facility / district level
Be aware of resources available for vaccination information in routine immunization
Note that we’re talking about paper and pencil forms here, not computer systems.

3. Overview of M&E framework for RI
Inputs
Outcome
Impact
Short-Term
Long-Term
Vaccines,
services delivered, advocacy, planning, training
Increase vaccination coverage
Reduce vaccine-preventable disease (VPD) burden
Reduce infant and child morbidity and mortality
In every country:
at least 90% fully immunized, with at least 80% in every district*
-Eradicate polio
-Reduce VPD mortality and morbidity
Reduce child mortality by 2/3 by 2015 (from 1990 levels)**
Reminder that reducing polio remains the top priority for STOP teams
Where and when appropriate, look for ways to support larger routine immunization infrastructure.
Goals
* WHO / UNICEF Global Immunization Vision and Strategy
** Millennium Development Goal 4

4. How monitoring works for RI
RI staff at all levels routinely monitor system performance and use this information to drive decision-making
Monitoring indicators include
Doses administered, coverage, dropout
Monitoring tools include
Monitoring charts, tables, monthly summary forms
Internal monitoring activities include
Supervision visits
Rapid community assessments

5. How evaluation works for RI
Internal evaluation activities include
Annual reporting of ~40 EPI indicators to WHO/UNICEF including doses administered and immunization coverage
Data quality self-assessments
Lot quality assurance surveys
External evaluation activities include
External immunization coverage surveys
Data quality audits

6. How Data for Action works for EPI
RI staff are expected to monitor their system AND use this information for action
Use of RI information to improve coverage is often the weakest part of RI
Supervisors (& you) should observe and ask staff if they know how to both calculate and use monitoring indicators, tools and activities

7. Common RI Indicators Service performance Access Utilization
Defined as number of children receiving one of the later vaccines
Coverage with DTP3 (DTP = DPT = DTC)
Coverage with measles
Percentage of children never reached
Access
Defined as availability/start of services (number of children who received one of the earliest vaccines)
Coverage with BCG or DTP1
Utilization
Defined as continued use of services
“Dropout rate” e.g. the number of children who received an early vaccination (DTP1) compared to the number of children who received a later vaccination (DTP3)

8. How to calculate coverage
Coverage formula
________number of doses given_______ X 100
estimated number of age-eligible children
in catchment area
Sources of error for coverage
Number of doses given
Estimated number of age-eligible children in the catchment area

9. How to calculate dropout
Dropout definition: Child begins but does not complete immunization schedule
Drop-out rate formula
[ ( # Beginning – # Ending ) / # Beginning ] X 100
Usually, series begins with DTP1 or BCG
and ends with DTP3 or measles
= [( DTP1 – measles ) / DTP1] X 100
= [( DTP1 – DTP3 ) / DTP1] X 100
= [( BCG – measles ) / BCG] X 100
= [( BCG – DTP3 ) / BCG] X 100

10. Visualization of Never-reached versus Drop-outs

11. Practice: Calculate Coverage and Dropout
 Compile population and coverage
data from previous 12 months
Problems analysis
Name of Health Facilities
Target pop < 1 yr
Doses given
Immunization coverage (%)
Infants non immunised
Drop out rate (%)
Identification of problems
Category
DTP1
DTP3
Measles
DTP1 & DTP3
Access
Utilization
1, 2,
3, 4 A B C D E F G H I J K L M N #
(C/B)x100
(D/B)x100
(E/B)x100
B - D
B - E
(C-D)/C x 100
See flow chart "Interpretation"
Berma
430
502
472
445
Dokuy
781
836
353
538
Dara
490
355
342
524
Ira
916
651
573
572
TOTAL

12. Tool 1: Monitoring chart
Used at multiple levels to visually track monthly performance
Charts include:
Number of doses administered per month
Cumulative number of doses administered in year
Coverage rate
Difference in number of doses administered between 2 vaccines (dropout)
Monthly target and cumulative target population
Good charts are:
Up to date
Properly filled

14. Completing an Immunization Monitoring Chart
Enter monthly objectives on the y-axis (target population ÷ 12)
Diagonal line represents regular progress over a 12 month period ending with 100% coverage
Every month
Enter the antigen total for the month
Calculate the cumulative total for the antigen
Mark the cumulative total on the chart with a dot
Calculate the cumulative number of drop outs
Calculate the cumulative drop out rate

15. Practice: How to fill in a monitoring chart
Zorgho, 2003
Target population < 1 = 12,897
DTP1
DTP3
Jan
836
525
Feb
909
631
Mar
818
539
Apr
1027
729
May
728
Jun
660
529
Jul
1001
650
Aug
860
562
Handout 10

16. Zorgho
2003
12897
1075
12897
11822
10748
9673
8598
7523
6449
5374
4299
3224
2150
1075

17. Zorgho
2003
12897
1075
12897
11822
10748
9673
8598
7523
6449
5374
4299
3224
2150
1075
836
836
909
1745
818
2563
525
525
631
1156
539
1695

18. Zorgho
2003
12897
11822
10748
9673
8598
7523
6449
5374
4299
3224
DTP1
2150
1075
DTP3
836
836
909
1745
818
2563
525
525
631
1156
539
1695
311
589
868 37 34 34

19. Tool: Immunization summary form
Facility summary form
Summary of doses administered at facility
Data summarized from session tally sheets
District summary form
Summary of doses administered across all facilities in district
Data summarized from facility summary sheets
Sent monthly by each level
Used nationally as source of information on number of doses administered

20. Illustrative flow of immunization summary information
National
Department of
Epidemiology
EPI program
Consolidated monthly
Surveillance report
Weekly reports
Consolidated monthly
reports, vaccination
Case reports
District
Monthly reports
Case reports
Weekly reports
Health Center
consolidates
Tally sheets
Patient registers
Case reports
Health Center
Health posts
Outreach sites

22. Liberia Immunization Summary Form

23. Tool: Defaulter Tracking System
Method to identify children who did not return for vaccination (drop outs)
Multiple tools used as “tracking system”
Ledger of children’s names who dropped out
Tickler box or blanket
Copy of vaccination card is kept with the month when child is expected to return
At end of month, remaining cards used to identify and then find children who did not return

25. Defaulter ledger example

26. Categorize the problem
Tool: Prioritize your locations by severity of problems (I)
What proportion of the children have ACCESS to immunization services? (what is DTP1 coverage?)
HIGH coverage with DTP1 (>80%)
Low coverage with DTP1 (<80%)
What proportion of children COMPLETE the immunization schedule (what are the DTP1 - DTP3 drop-out rates?)
Drop-out rate <10%
Drop-out rate >10%
Drop-out rate <10%
Drop-out rate >10%
Categorize the problem
Dropout rates are low = good utilization
Coverage is high = good access
Category 1 (no problem)
Dropout rates are high= poor utilization
Coverage is high = good access
Problem Category 2
Dropout rates are low = good utilization
Coverage is low= good access
Problem Category 3
Dropout rates are high = poor utilization
Coverage is low= poor access
Problem Category 4
Source: Increasing immunization coverage at the health facility level, WHO 2001

27. Tool: Prioritize your locations by severity of problems (II)
 Compile population and coverage
data from previous 12 months
Problems analysis
Name of Health Facilities
Target pop
< 1 yr
Doses given
Immunization coverage (%)
Infants non immunised
Drop out rate (%)
Identification of problems
Category
Priority
DTP1
DTP3
Measles
DTP1 & DTP3
Access
Utilization
1, 2, 3, 4 A B C D E F G H I J K L M N O #
(C/B)x100
(D/B)x100
(E/B)x100
B - D
B - E
(C-D)/C x 100
See flow chart "Interpretation"
Berma
430
502
472
445
117%
110%
103% 30
-15 6% 1  3
Dokuy
781
836
353
538
107%
45%
69%
483
243
58% X 2  1
Dara
490
355
342
524
72%
70% 13
-34
3.7% 3  2
Ira
916
651
573
572
71%
62% 78
344
12% 4  4

28. Tool: Rapid community assessment
Monitoring tool used by supervisors to rapidly understand whether vaccination services are reaching the community and identify barriers to vaccination
Not a survey and does not generate coverage rates
Useful to use in places where administrative vaccination coverage is reportedly high
A rapid assessment should not find many unvaccinated children
General method
Select ~20 households and assess vaccination status and reasons for no vaccination
Should be conducted in parts of a catchment area where low coverage is expected

29. Thanks
Questions?