Institute for International Programs An international evaluation consortium Institute for International Programs An international evaluation consortium A common evaluation framework for the scale-up to achieve the health MDGs International Evaluation Consortium Institute for International Programs, Johns Hopkins Bloomberg School of Public Health May 2008 Including the results framework for maternal, newborn and child health

Presentation Outline 1.Background, rationale, methods and partners 2.The strategic framework 3.The results framework for maternal and child health and nutrition 4.Evaluation challenges

BACKGROUND, RATIONALE, METHODS AND PARTNERS Section 1:

Rationale for a common framework  To support the comparison of results and costs within and across countries that may use different approaches to implementation  To promote systematic attention to the methodological challenges of evaluating large- scale approaches  To inform pay-for-performance approaches Victora CG, Black RE, Bryce J. Learning from new initiatives in maternal and child health. Lancet 2007; 370:

What is included in an evaluation framework? 1.A strategic framework including general principles 2. A results framework that includes:  A conceptual model specifying how activities will lead to outcomes and impact  A set of compatible designs for evaluation of country-level initiatives, to allow comparisons across places and time  A set of common indicators and other measures

Framework development process Global business plan Global campaign IHP+ Catalytic initiative BMGF/PMNCH Rapid Scale Up Doris Duke MCH results framework Overall strategic framework Common evaluation framework Initiatives Evaluators & products

THE COMMON STRATEGIC FRAMEWORK Section 2:

Framework Terminology TermDefinition Evaluationrigorous, science-based collection of information about program activities, characteristics, outcomes and impact that determines the merit or worth of a specific program or intervention Monitoringroutine tracking and reporting of priority information about a program and its intended outputs and outcomes…to assess whether resources are spent according to plan and whether the program is resulting in the expected outputs Monitoring performance project or programme monitoring which aims to provide feedback for improved performance/ implementation Performance- based funding provision of financial incentives based on measured progress Data quality audits a method to assess recording and reporting systems in the context of performance-based funding Adapted from Boerma T, Bos E, Walford V et al. International Health Partnership+. A common framework for monitoring performance and evaluation of the scale up for better health. Draft 4, February 2008.

Principles Collective action: Primary focus on the contribution of the collective efforts to scale-up the health sector response in countries Alignment with country processes: build upon national processes that countries have established to M&E progress in the implementation of national plans Balance between country participation and independence: driven by country needs but conducted in a manner which maintains independence of evaluation Harmonised approaches: common protocols and standardized outcome indicators and measurement tools, with appropriate country adaptations Capacity building and health information system strengthening: systematic involvement of country institutions Adequate funding: between 5% and 10% of the overall scale-up funds set aside for monitoring performance, evaluation, operational research and strengthening health information systems Adapted from Boerma T, Bos E, Walford V et al. International Health Partnership+. A common framework for monitoring performance and evaluation of the scale up for better health. Draft 4, February 2008.

Inputs Outputs Outcomes Impact Funding Domestic sources International sources Plan Coherent, prioritised and funded Harmonization Aligned international efforts with national plan Well coordinated and harmonized support National plan implementation Systems strengthening Priority interventions scale-up Capacity building Programmes Institutions People Accountability Performance monitoring Results focus and evaluation Use for better practices Health system strengthened Governance, HR, medical products, information Increased service utilization and intervention coverage Reduced inequity (e.g. gender, socio- economic position) Responsiveness No drop-off non- health sector interventions (e.g. water & sanitation) Improved survival Child mortality Maternal mortality Adult mortality due to infectious diseases Improved nutrition Children Pregnant women Reduced morbidity HIV, TB, malaria, repr. health Improved equity Social and financial risk protection Reduced impoverishment due to health expenditures Improved services Access, safety, quality, efficiency Process Aid process monitoring Resource tracking Strengthen country health information systems Evaluation: process, health systems strengthening, impact M & E action Health system monitoringCoverage monitoringImpact monitoring Implementation Monitoring Strategic Framework Adapted from Boerma T, Bos E, Walford V et al. International Health Partnership+. A common framework for monitoring performance and evaluation of the scale up for better health. Draft 4, February 2008.

THE RESULTS FRAMEWORK FOR MATERNAL, NEWBORN AND CHILD SURVIVAL Section 3:

Three components of a results framework 1. A conceptual model specifying how activities will lead to outcomes and impact 2. A set of compatible designs for evaluation of country-level initiatives, to allow comparisons across places and time 3. A list of common indicators and other measures

What is a conceptual model in the context of a results framework? A description of the activities and pathways that will lead from program inputs to impact on health and nutrition Inputs Funding Planning & policies Harmonization & efficiency Inputs Funding Planning & policies Harmonization & efficiency Outputs Health services delivery Quality Behavioural Interventions & knowledge Outputs Health services delivery Quality Behavioural Interventions & knowledge Outcomes Service utilization and intervention coverage Behavioural change Reduced inequity Outcomes Service utilization and intervention coverage Behavioural change Reduced inequity Impact Mortality Morbidity Nutrition Impact Mortality Morbidity Nutrition Process Training & Capacity building Procurement and supply Guidelines IEC Community mobilization Process Training & Capacity building Procurement and supply Guidelines IEC Community mobilization A generic conceptual model based on the strategic framework

Why is a conceptual model essential in a results framework?  To clarify expectations of program planners/ developers  To define the evaluation questions and select indicators  To support the design and estimate sample sizes  To guide analysis and attribution of results  To compare and interpret results across sites  To track changes in assumptions as they evolve in response to evaluation findings  To stay honest about what was expected The generic conceptual model must be adapted to reflect the plans and expectations of each country/program.

Common elements of the conceptual model for the global scale-up  MORBIDITY AND MORTALITY High and equitable coverage with proven interventions Country-specific implementation & health system strengthening Inputs Outputs Outcomes Impact Process

Designs linked to national monitoring The evaluation framework must be implemented in ways that build institutional capacity for evaluation in countries. National program monitoring Impact evaluation Policies/plans/ resources Implementation (provision & quality) OutcomesImpact

Evaluation challenges  Complete tracking of program and contextual factors  Full consideration of equity  Evaluating “scaling-up” and “catalytic actions”  Measuring “community engagement”  Assessing “health system strength”  Capturing changes in mortality within short time frames – new approaches needed

Summary The common evaluation framework for the global scale-up to the health MDGs  There is a common evaluation framework; all countries and partners should use it as a starting point for planning their evaluations  Main purpose of evaluating global scale up is assessment of program effectiveness  Requires comprehensive prospective evaluation designs  Complementary evaluation goals of independence and capacity building  Key constructs need further definition (e.g., “scaling-up”; “catalytic”, “health system strength”)