1. Overview of Steps 1-4 of Data Analysis
ROUTINE HEALTH INFORMATION SYSTEMS
A Curriculum on Basic Concepts and Practice
MODULE 5:
RHIS Data Analysis
SESSION 2:
Overview of Steps 1-4 of Data Analysis
The complete RHIS curriculum is available here: routine-health-information-systems/rhis-curriculum

2. Learning Objectives and Topics Covered
By the end of this session, participants will be able to:
Select appropriate indicators for data analysis
Conduct a basic desk review of data quality and adjust data if necessary
Select appropriate denominators
Compare findings from routine data with findings from other data sources
Analyze routine data to produce information products (tables, graphs, and maps)
Topics Covered
Selection of indicators for analysis
Desk review of data completeness and internal consistency
Selection of appropriate denominators
Comparison of findings from routine data with findings from other data sources

3. Select a limited set of core indicators Review data quality
5 Steps of Data Analysis
Select a limited set of core indicators
Review data quality
Select appropriate denominators
Reconcile findings with estimates from other data sources
Communicate key findings (to be discussed in Module 5, Session 3)
5 steps applied at district level:
Select indicators for monitoring and evaluation (M&E) of the district health system.
Data quality review: Regularly review facility-level data for completeness and internal consistency.
Denominators: Consider using ANC1 and/or DTP1 as denominators for facility-level estimates and district-level estimates.
Reconciliation with other data sources: Surveys don’t provide district estimates but regional estimates can provide approximations of the district estimate.
Communication: Give feedback on findings to health facilities to promote improved data quality and use (to be discussed in session 3).
The fourth item does not necessarily apply in all circumstances.

4. S1: First Step
Select a Limited Set of Core Indicators

5. What Makes an Indicator “SMART”? (Characteristics of Good Indicators)
Specific: Indicator is concrete, detailed, focused, and well-defined
Measurable: Indicator tells how many or how much and can be measured with identified measurement sources
Agreed upon: Stakeholders vested in a specific M&E question should agree that indicator is relevant
Relevant: Indicator generates data that can answer the question of interest
Time-bound: Indicator specifies time frame of what it is measuring
[Remind the participants that this slide was presented in Module 2, Session 1.]

6. S1: Select a Limited Set of Core Indicators
Core indicators should:
Be based on M&E framework for the national health sector strategy
Be programmatically relevant, and facilitate program management
Reliably and comprehensively assess the performance of the health system (whether national or subnational levels)
Be clearly defined
Have numerators that are measurable with routine health data

7. S1: Core Indicator with 2 Data Sources: DTP3
Routine health information systems
Numerator: Number of infants immunized with DTP3 by 12 months of age in a given year
Denominator: Total number of surviving infants <12 months of age in same year
Population-based survey
Numerator: Number of children ages 12–23 months who received three doses of DTP3 vaccine by age 12 months
Denominator: Total number of children ages 12–23 months surveyed

8. S1: Core Indicators Measured Reliably with One Data Source
Antiretroviral therapy (ART) retention rate
Source: Routine health information systems
Numerator: Number of adults and children with HIV, alive and on antiretroviral therapy (ART) 12, 24, 36 months (etc.) after initiating treatment Denominator: Total number of patients initiating ART during a specific period
Fully-immunized child
Source: Population-based survey
Numerator: Number of children ages 12–23 months who received 3 doses of OPV, 3 doses of DTP, and 1 dose each of BCG and measles vaccine before age 12 months
Denominator: Total number of children ages 12–23 months surveyed
Identify the best data source for different types of core indicators.

9. Some Indicators Are Not Clearly Defined or Cannot Be Reliably Measured with Routine Data
Percentage of health facilities with a skilled provider
Percentage of health facilities with an adequate supply of drugs
Some health indicators can be reliably measured with a household survey or health-facility survey but not with data reported routinely by health facilities:
Proportion of population using an improved drinking water source
Proportion of health facilities with treatment guidelines
Infant mortality rate
Some indicators are not clearly defined and usually cannot be reliably measured with data reported routinely by health facilities:
% of health facilities with health equipment

10. Practicing the First Step
Exercise: Part 1
Review the list of core health indicators (on the next slide (Handout 5.2.1)
Identify which indicators can/cannot be reliably measured with routine data
Can the numerator be defined with routine data?
Do you need additional data sources to measure the indicator?

11. List of Health Indicators
Percentage of pregnant women attending antenatal clinics who are screened for syphilis
Neonatal mortality rate
Number of neonatal tetanus cases
Cost of one month’s supply of contraception as a percentage of monthly wages
Percentage of infants born protected against neonatal tetanus in a specified period
Measles vaccine coverage rate
Percentage of registered new and relapse tuberculosis (TB) patients with documented HIV status
Percentage of children ages 12–59 months who were dewormed in the past six months
Percentage of HIV-positive infants born to HIV-positive women
Maternal mortality ratio
Number of health facilities providing comprehensive emergency obstetric care functions per 500,000 population
Exclusive breastfeeding rate
TB treatment success rate
Percentage of health facilities with systems that support quality service delivery
Percentage of districts with current trend analysis for selected priority diseases at a given time (e.g., month)

12. S1: Practicing the First Step
Exercise, Part 2
Review the WHO’s Global Reference List of Core Health Indicators (Handout 5.2.3) or the list of standard indicators in Handout
Select five indicators from the list.
For each indicator:
Specify the numerator
Specify the denominator
What is the data source for the numerator?
What is the data source for the denominator?
How can the indicator be interpreted?
Ask participants to review the list of WHO indicators and identify those that are not well-defined and/or not reliably measureable with routine data.

13. S2: Second Step
Review Data Quality

14. Step 2: Review Data Quality (see also Module 4)
Completeness and timeliness
Completeness of reports
Completeness of data
Timeliness of reports
Internal consistency
Accuracy
Outliers
Trends
Consistency among indicators
External consistency
Data triangulation
Comparison with data surveys
Consistency of population trends
External comparisons (population denominators)
Facilitator should highlight here that the data quality was covered on Module 4 and participants are just being reminded that when it comes to data analysis, we need to make sure that the data being analyzed are reviewed for quality beforehand.
Go quickly over the slide and stress the following:
Accuracy: Measured against a reference and found to be correct
Completeness: Present, available, and usable
Timely: Up-to-date and available on time

15. S2: Demonstration, Excel
How to use Excel to:
Create a chart
Participants are invited to practice simultaneously.
Here are two websites with 5-minute videos explaining how to create a chart using Excel:
create-a-simple-chart-with-two-clicks.html

16. S2: Demonstration DHIS 2
How to use DHIS 2 to:
Verify the definition of an indicator
Numerator and denominator
Create a pivot table
Create a chart
Participants are invited to practice simultaneously.

17. S3: Third Step
Select Appropriate Denominators

18. S3: How Do We Get Coverage Denominators?
First, estimate the size of the target population.
Common target populations for health-facility-based indicators:
Total population, children< 5 years, infants, pregnancies, women of reproductive age, live births at health facilities
Size of target populations is often estimated (such as projections/modeled estimates from national population census).
Limitations of estimates:
Reliability declines with years since last census
Internal migration may make estimates of populations of regions and districts unreliable
Some programs may use their own denominators.

19. How Do We Get Denominators?
Document how the denominator value was obtained:
Methods and assumptions used to calculate the denominator
Annual rate of growth if denominators are based on projections of census figures
Present these along with rest of analysis
If service coverage >= 95% and data are of high quality, use ANC1 or DTP1 to estimate the number of surviving infants.
Use of service statistics to estimate size of target population can modify conclusions reached about which districts are strong performers and which are weak performers.

20. Estimating Denominators
Estimating the number of surviving infants:  
Total population: 5,500,000
Crude birth rate (CBR): 30/1,000
Infant mortality rate (IMR): 80/1,000
Number of surviving infants
Total population x crude birth rate x (1 - IMR)
= 5,500,000 x 30/1000 x ( )
= 5,500,000 x x 0.920
= 151,800
Note that an alternative method that is sometimes used for estimation of surviving infants is to project the number of infants counted during the most recent census. Infants are often under-counted during censuses, so it is usually preferable to use census data to estimate the CBR than to use the CBR to estimate births, surviving infants, and pregnancies.

21. Data Quality Check for Denominators
Pregnancies = births + pregnancy loss (2% to 10%);
Surviving infants = births – infant mortality
Number of pregnancies, deliveries, live births, infants Tanzania district example, 2014
Estimates of pregnancies, deliveries, births, and surviving infants must be internally consistent.
The denominators used to calculate coverage with ANC services, delivery at health facilities, and immunization must be internally consistent.
Due to early pregnancy loss, the number of early pregnancies (e.g., if measuring coverage with ANC care before 12 weeks) should be about 10% greater than the number of births.
Due to stillbirths, the number of late pregnancies should be about 2% greater than the number of births.
Due to births of twins, the number of deliveries may be 1% less than the number of births.
Due to infant mortality, the number of surviving infants is less than the number of births.
Whatever assumptions are made:
The estimates of pregnancies, deliveries, births, and surviving infants must be consistent with one another.
At the national level, no indicator should have a coverage > 100%.
You must describe your assumptions as part of the analytic report.

22. S4: Fourth Step
Reconcile Findings with Estimates from Other Data Sources

23. S4: Reconcile Findings with Estimates from Other Data Sources
Compare data from parallel systems that routinely report the same health events.
Compare estimates from routine health-facility data with estimates from household surveys at the national and regional levels.
Compare available data with statistics that have been officially reported to WHO.

24. S4: Estimates from Two Sources of Routine Health Facility Data
Administrative estimates of 2014 DTP3 coverage, by district,
DPI/JRF versus HMIS/DHIS data
Note: Red marks are for districts with a negative dropout rate, according to EPI/JRF data.

25. S4: Compare Estimates from Household Surveys
DTP3 immunization coverage, Tanzania, 2009–2012
Sources: Routine EPI facility data, 2010 DHS, and 2011 Immunization Coverage Survey

26. Reconciling with Survey Findings Is Not Always Easy
2014 WHO-UNICEF report on trends in DTP3 coverage in Ethiopia
Administrative/official estimates (red stars and circles) versus surveys (vertical red lines) versus WHO-UNICEF estimate (blue line)
Administrative coverage estimates (the red asterisks) have varied considerably from findings of most coverage surveys. The surveys, even though sometimes conducted in consecutive years, yielded markedly different estimates.
Surveys are often considered to be the gold standard for measurement. Yet, the quality of survey data depends on such things as the percentage of children for whom immunization cards were observed. In the case of these surveys in Ethiopia, cards were observed for as few as 29% of children.

27. Reconciling with Survey Findings Is Not Always Easy
2014 WHO-UNICEF report on trends in DTP3 coverage in India
Administrative/official estimates (red stars and circles) versus surveys (vertical red lines) versus WHO-UNICEF estimate (blue line)
Administrative coverage estimates (the red asterisks) have varied considerably from findings of most coverage surveys. The surveys, conducted in consecutive years, have yielded similar estimates.

28. Small Group Exercise (Steps 1-4 )
Distribute Handouts 5.2.4a and b.
Form small groups of 4–5 participants.
Using Excel and the spreadsheet provided (Penta, 2014 data from Tanzania, Handout 5.2.4B), calculate indicator values by region for:
Penta1 coverage rate
Penta3 coverage rate
Penta1-Penta3 dropout rate
For each region, specify whether access is good or poor.
For each region, specify whether utilization is good or poor.
Categorize the immunization problem in each region (if any).
Brainstorm the differences in coverage between regions.
Discuss what action managers can take if coverage and dropout rates indicate problems.

29. P2: Small Group Exercise, Option B (DHIS 2)
Distribute Handout
Form small groups of 4–5 participants.
Choose either ANC4 or DTP3 as your indicator.
Using DHIS 2, verify the definition of the indicator.
Create a pivot table with indicator values by district.
Graph trends in the completeness of reporting by year for the past four years by:
Facility (hospital versus health center) and
Managing authority (public versus private)
Discuss how the trend in completeness would affect the apparent trend in the indicator.

30. ROUTINE HEALTH INFORMATION SYSTEMS
A Curriculum on Basic Concepts and Practice
This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government.