A Comparison of Different Methods of Data Collection Health System Innovations Workshop Abuja, Jan , 2010

Three Pillars of M&E 1.Setting Objectives and selecting Indicators √ 2.Collecting data using appropriate methods 3.Using the data to make decisions and improve performance 2

Important Methods Different methods of data collection required, each with benefits & issues 1.Routine reporting system (HMIS): reported data coming from service providers 2.Household surveys (HHS): statistical sample of households in the community 3.Health Facility Assessments (HFA): a survey of a sample of health facilities to assess QOC 4.Quantitative supervisory checklist 3

1. Routine Reporting - HMIS Advantages Usually near real time Highly Disaggregated Local managers can their manage own performance Disadvantages often inaccurate service providers have incentive to overstate difficult to provide data on equity, quality can’t assess effects at community level Expensive (HW time) 8x cost of HHS Misses info from private sector 4

Excessive & Exclusive Reliance on HMIS is Fraught with Danger Murray et al have found that GAVI ISS may have over-paid by 93% Data Quality Audit (DQA) was no help in discovering discrepancies between HMIS & surveys People want to tell their superiors good news May worsen when tied to some form of bonus 5

HMIS vs. Household Survey DPT3 Coverage in Pakistan Average difference = 14% points 6

2. Household Surveys Advantages More accurate Can be independent of service providers Provides data on coverage, equity, expenditure, CPR, ITN use, satisfaction, etc. Disadvantages Relatively expensive (cheaper than HMIS but an incremental cost) Episodic, not done in real time Level of disaggregation 7

M&E: More Frequent Household Surveys In a rapidly changing system need more frequent data Data required to verify HMIS Lots of data best obtained from HHS – Household behaviors (use of ITNs, CPR, etc.) – Utilization, patient satisfaction, payments – Knowledge and behaviors – Source of care (private, public, non-qualified) Suggestion: Every 2 years with disaggregation (LQAS or cluster surveys at state level) 8

3. Health Facility Assessments Advantages Can assess quality of care, more than anecdotes Can be independent of service providers Can be done more often than HHS Relatively inexpensive Disadvantages Complex to design Lots of data, can overwhelm managers Cannot provide information on coverage, equity 9

4. Quantitative Supervisory Checklist A reduced version of a health facility assessment Objectively assesses a variety of indicators to come up with total score. Takes about 1-3 hours to complete A copy of results left in the health facility, easy to track progress QSC is both a management intervention and tool for M&E 10

4. Quantified Supervisory Checklists Advantages Can assess QOC. Can be independent of service providers Can be done often Inexpensive Clarifies what is expected of HWs Can be adapted to conditions as they change Disadvantages Challenging to design Cannot provide information on coverage, equity Ensuring continued use is difficult 11

Recommendations Use all four approaches – none are perfect Relying excessively on HMIS is unwise Allocate sufficient budget to carry out various approaches to data collection (5-7% of projects) Make M&E somebody’s job 12