Framework for evaluation of scaling up of programmes Consultative workshop on Health Impact Accounts Brussels 6 February 2012

Demand for health impact evaluation MDG 2015: accurate measurement of child and maternal mortality GAVI: immunization coverage  immunity  incidence and mortality reductions Global Fund: coverage of interventions (ART, bednets and malaria treatment, TB treatment),  impact on morbidity & mortality PEPFAR: new infections (children, adults), mortality NCD political declaration: targets on mortality due to major chronic diseases, risk factor prevalence Commission on Information & Accountability for Women's and Children's Health: birth and death registration Post 2015 development agenda: place of health as intervention; health as an outcome measure

Monitor Action Review COUNTRY GLOBAL Review Action GLOBAL Monitor Accountability

Evaluation = Accountability? Accountability: was the money well spent? Evaluation also means learning: which components of the programmes and interventions worked best? what are the implications for future programs? Current demand for results and relating those to investments and interventions: –global level: lives saved computations –large-scale effectiveness evaluation at country level Supply: –Somewhat more investment in evaluation –More retrospective and to a lesser extent prospective evaluations

Evaluation of effectiveness of large-scale programmes More surveys to ascertain health impact –India: conditional cash transfer program for facility births, using District Level Household Surveys (Lim et al., 2010): coverage, not maternal mortality –West Africa: child survival program (Bryce et al. 2010): used districts, DHS –Indonesia: World Bank MCH interventions (Baird et al., 2011): provinces 15 yr –Issues: impact measurement, measurement of intervention implementation strength, attribution, no counterfactual, time lag Large scale evaluations of complex interventions –Global Fund five year evaluation: retrospective, multiple diseases, multiple data sources, multiple countries, short time period, heavy reliance on modelling, generated no major new evidence of impact –Avahan HIV prevention: $27 mln investment in evaluation, prospective (2003), multiple data sources; dozens of publications (STI, BMC Public Health); lives saved based on money disbursed and HIV antenatal surveillance in districts (Ng et al., 2011): 100,000 in 5 years (25, ,000) –GAVI: prospective evaluation investment decision board

6 |6 | Country accountability processes in context of National Health Strategies

Country-led platform for information & accountability Monitoring & review of the national health strategy Country data generation & compilation Monitoring reports Global Fund reporting GAVI reporting MDG / UN reporting PEPFAR reporting Programme Reporting (TB, MCH, HIV, etc.) Country data information generation & compilation Analysis & synthesis Statistical reports Programme reports Harmonization of reporting requirements Minimization of reporting requirements External validation and estimatesCommon standards and tools Independent reviews Data quality assessment Evaluation Reviews

Technical framework for M&E and review of National health strategies Data collection Indicator domains Analysis & synthesis Communication & use Administrative sources Financial tracking system; NHA Databases and records: HR, infrastructure, medicines etc. Policy data Facility assessments Service readiness Population-based surveys Coverage, health status, equity, risk protection, responsiveness Clinical reporting systems quality, coverage, health status Civil registration Data quality assessment; Synthesis of progress and performance; Evaluation Regular country health sector review processes; Global reporting Improved health outcomes & equity Social and financial risk protection Responsiveness Efficiency Financing Infrastructure; ICT Health workforce Supply chain Information Intervention access & services readiness Intervention quality, safety Coverage of interventions Prevalence risk behaviours & factors Governance Inputs & processesOutputsOutcomesImpact

9 |9 | National evaluation platform (Victora et al. 2010) Traditional evaluation designs do not work for large scale effectiveness evaluation: no comparison population, counterfactual difficult, ethical reasons Prospective design, linked with national monitoring efforts Use districts as the unit of design and analysis – Contextual information – Ensure continuous monitoring of input, process and output indicators Collect additional data before during and after the evaluation period using multiple methods – Baseline (and intermediate) survey, coverage, adequate sample size for district analysis – End survey: include retrospective mortality data (expand planned surveys such as DHS if possible) Use multiple data analysis techniques to deal with data gaps and biases – Areas with and without programmes (before-after analysis with comparison group); stepped wedge design (if sequential roll-out); dose response analyses Include interim (formative) and summative analyses Promote country ownership, transparency and donor coordination, while maintaining a rigorous of the cost effectiveness of different scale-up approaches Source: Victora CG, Black RE, Boerma JT, Bryce J. Measuring impact in the MDG era and beyond: A new approach to large-scale effectiveness evaluations. Lancet, published on line 9 July 2010.

10 | Comprehensive analysis of country health progress and performance A stepwise approach Improved health outcomes & equity Financial risk protection Responsiveness Financing Infrastructure / ICT Health workforce Supply chain Information Intervention access & services readiness Intervention quality, safety and efficiency Coverage of interventions Prevalence risk behaviours & factors Governance Inputs & processesOutputsOutcomesImpact Have finances been disbursed? Have policies been changed? Is the process of implementation happening as planned? Has access to services improved? Did the quality of services improve? Has utilization improved? Did intervention coverage improve? Have risk behaviours improved? Have health outcomes and equity improved? Are services responsive to the needs? Are people protected against financial risks? Contextual changes Non health system determinants Contextual changes Non health system determinants

Evaluation platform Advantages Adapted to current reality of multiple simultaneous programs/interventions Integrated with country health system Promotes country ownership and donor coordination Evaluation as a continuous process Flexible design allows for changes in implementation Limitations Observational design (but no other alternative is possible) High cost particularly due to large size of surveys Requires transparency and collaboration by multiple programs and agencies

Global evaluation framework Evaluation = accountability + learning Global computations only as good as country data inputs Critically dependent on investments in country platform for information & accountability Data availability (e.g. causes of death) and quality (e.g. program implementation strength) are key; (but more data may also imply more complicated answers) Risk of disconnect between global and country relevance: –global health impact accounts not relevant for countries –Computations likely to be complex and full of assumptions –Countries do not care about attribution to development partners