Measuring success for mHealth Lessons from monitoring and evaluation of Vodafone Foundation & UN Foundation’s mHealth program in Africa 28 October 2009 Andrew Stern – Partner, Dalberg Global Development Advisors

Background and summary 1 Dalberg has been working with UNF and Vodafone Foundation for two years to understand the impact of its mHealth program, with an in-depth focus on the programs in Senegal, Ethiopia, and Kenya The approach used a ‘Theory of Change’ approach to analyze and monitor activities, outputs, and outcomes that lead to impact Focusing on activities, outputs, and outcomes for monitoring & evaluation enables: –Identification of critical constraints to achieving impact and therefore an opportunity to develop mitigation approaches; –Focus on cost efficiency to yield greater program effectiveness; –Demonstration of relative effectiveness of a program across geographies Comparing across countries, major success drivers have been identified including: –The quality of data-driven analytical and decision-making processes to make use of the data being collected; –The fact that narrow but deep pilots often work better for demonstration; and, –How critical it is to ensure complementary inputs, such as transportation costs, to ensure collection of data

The UNF-VF partnership 2 Objective: Bring the latest technology solutions to UN field work, using sustainable, digital tools to save lives Solution:EpiSurveyor enables development of customizable surveys for cell phones and PDAs Partners: DataDyne, World Health Organization, Ministries of Health, UNF-VF Duration: EpiSurveyor is now active in 19 countries Since 2005, the Partnership has worked with DataDyne and WHO to develop and roll out EpiSurveyor

Partnership activities Decision-making and action to meet needs Analysis and information sharing EquipmentTraining Software Development Collection management and supervision Logistical support Customizable survey development Saving and improving lives Improved health systems and monitoring More effective campaigns (vaccinations, etc.) More effective outbreak response Impact ~3-5 years Outcome ~6 months Outputs ~3 months Activities ~3 months National Regional District Health data collection 3 Theory of change

4 Impact ~3-5 years Outcome ~6 months Outputs ~3 months Activities ~3 months Country health systems with more, better, and faster data for decisions and action Provide data collection method for as many health applications and health facilities as possible Do more with same funding, or less Reduce lives lost and improve lives Intent 10.Disability adjusted life years (DALYs) prevented 9.Percent of survey questions showing positive improvements from prior month 8.Percent of decision-making meetings that discuss the data Indicator 7.Records analyzed as % of records collected 6.Records collected as a % of visits conducted 5.Visits conducted as % of potential visits in program 4.Visits covered by program as % of all potential visits 3.% of Ministry of Health divisions using PDA 1. Cost per survey record collected Indicators and performance management

Focusing on measuring outputs and outcome allows for the identification and mitigation of critical challenges that constrain impact ILLUSTRATIVE Improved health systems and monitoring More effective campaigns (vaccinations, etc.) More effective outbreak response Outcome Outputs Decision- making and action to address needs Analysis and info sharing Health system supervisory data collection x Records may not be compiled Few records may be collected Some records may be lost May have insufficient analytical tools to do trend and other analyses Analytical needs at various levels may not be met May not have clear use of data by decision makers May not have clear tracking of decisions made with data xx 5

Country example: Senegal yielded a tangible example of impact 6 Context In Senegal, there is evidence that the process is working: data is collected, analyzed and then used as an input into decision making. Given that foundation, the team identified a specific instance where the deployment of the PDA and integrated survey can be reasonably linked to an improved health outcome. Story Initial PDA collected data showed low usage of the partogram, a simple form used by midwives to monitor the status of a delivery [at left]. Delivering babies without using the partogram increases the chances of complications. Seeing this gap in his district, the Pikine District Chief ordered a tripling of production of the forms and ensured they were distributed to midwives. Between March and August 2008, data collected with the PDA showed an increase in partogram usage in the pilot districts by 14 percentage points, even though usage remained unchanged outside those districts. Just one example The partogram case study is one example of estimated health impact. Two-thirds of the 82 questions tracked by EpiSurveyor showed improvement and increased by 10% or more. Additional case studies may be feasible based on previous data and continued support for data collection in the country. Case Study: Partogram Usage in Senegal