Health Impact Evaluation: An Introduction Temina Madon, PhD Executive Director Center of Evaluation for Global Action University of California, Berkeley

What is Impact Evaluation? Identify a health need or target. Proportion of children under 5 sleeping under ITN is < 6% and target is 60%. Health worker absenteeism is reducing patients’ use of clinics (unpredictable access to care is a deterrent) What must be done to meet needs or reach targets? Increase access to and proper use of ITNs Decrease health worker absenteeism What interventions or services could be used? Cost-free distribution of ITNs through schools, or door-to-door distribution. Incentive-based payments to health workers conditioned on health coverage or on attendance. Which is the most robust program to use? Pilot a few programs and measure their impacts. Is program A able to reliably produce desired results? What about program B? Use treatment and control groups to strip away alternative explanations for the results.

Why Impact Evaluation? If you need evidence that a program works Does the program actually improve health outcomes? Does it reach the people in need? Accountability to civil society Accountability to funders Ability to target a limited budget If you want to improve the program over time Results-based management Demonstrate cost-effectiveness If you want to scale up but need proof of concept

Objectives of this Workshop Understand how IE is related to M&E Identify questions suitable for impact evaluation Determine which indicators and designs to use Understand impact evaluation methods Develop an impact evaluation concept note for an intervention of your choice

M&E vs. IE

Are deworming treatments being delivered as planned? Does school-based delivery of deworming medication increase school attendance? What is the correlation between access to health clinics and the proportion of children under 5 receiving routine vaccinations? Do house-to-house immunizations lead to an increase in the proportion of children under 5 who are immunized, relative to the level in communities with annual vaccination campaigns? Which costs less per child vaccinated?  M&E  IE  M&E  IE

M&E vs. IE M&E can often be started well after the program has been designed and implemented IE generally needs to be considered at the outset of a program’s implementation or scale-up, because you need to build the evaluation into the design of the program

M&E vs. IE M&E allows you to guess how the program may have hurt or helped people. IE allows you to measure the program’s effects directly. You will rule out every other explanation for the observed effects.

What to Evaluate? A health service program –ITN distribution at pre-natal clinics –Oral rehydration packets and education for new mothers –Deworming medicines in schools Health systems project –Health insurance plans –Clinic staff retention Local or federal health policy –Seatbelts law –Tobacco tarriffs Behavior Change –Patient adherence to treatment –Improved care provided by health care provider –Community-wide adoption of prevention practices Services & Systems –Quality of Services –Cost-Effectiveness Health Outcomes –Maternal mortality –Child survival

When to use IE? M&E can be used to fine tune a program’s design, but IE will: –Tell you whether or not the program is “worth the money” (Does it work?) –Guide results-based management –Improve communication with the public & civil society