Introduction to Comprehensive Evaluation Justice Nonvignon, PhD School of Public Health, University of Ghana, Legon July 2016 International Workshop on Impact Evaluation of Population, Health and Nutrition Programs

Acknowledgments Sangeeta Tikyani Singh and Juan Pablo Gutierrez (INSP Mexico) contributed to this presentation. Materials were adapted from MEASURE Evaluation and INSP Mexico materials.

Comprehensive: One Word, Multiple Potential Meanings Regarding Evaluation Comprehensive in terms of scope (i.e., evaluating a sector, broad policy, government) Comprehensive in terms of approach (i.e., multiple perspectives) Comprehensive in terms of methods Comprehensive in terms of the evaluatee (programme, policy, intervention, institution)

What kind of comprehensiveness is desirable? What is relevant for the evaluatee? Depends on where in the cycle of operation How better address relevant questions? Perspective, tools, methods that are most suitable for a particular question

What are policies meant for? Ideally: Decision making to improve welfare Identify a condition that requires attention Design interventions/ programs / policies Implement interventions/ programs / policies What to do and how to do it? Planning Real life It is important to contextualize this evaluation view. This has been developed thinking on public programes and policies, and under the general view that decision making in the public context is related to improving welfare. So, when a condition that is decided that requieres attention is identified, there is design process that then will be follow by the implementation. This is intendent to translate what was planned to the real life

Before effectiveness: design & operation Identified causes Best available evidence Operation Implementation according to design Standardization Results Accomplishment of goals Effectiveness So, in our context, that most of the focus is put on final results, i.e. the program is reaching what was intended to or not, we wanted to highlight that before thoses results can or cannot be reached, it is very important to discuss design and operation

Why evaluate? Decide allocation Learning Improve performance Evidence-based policy: Improving efficiency Improving effectiveness Why evaluate? Decide allocation Provide evidence of destination and results of public funds Accountability Learning I am sure that this is something that you have your own way to present it, but this has been useful for us in thinking the different questions an evaluator may answer. We evaluate, of course, because we want to learn, as we are an academic institution. But the demand for evaluation is driven by the need for useful information for the programs. Policies and programs are evaluated because there is a need to generate evidence to inform the decision making process. We promote evaluation because we are also promoting evidence based policy and because we are convinced of the relevance of evaluation for the accountability process. Identify best practices and opportunity areas Test alternatives within existent programs Transferability to other contexts Modifications to existent programs. Assessment of continuation. Improve performance

Comprehensive evaluation

A general framework to connect different types of evaluations Basics of comprehensive evaluation Evaluation phases mirroring programs life-cycle Linking program phases from evaluation perspective It is proposing evaluation as a stepwise building The design of a program is its foundations, identify what is intended to achieve Define what indicators should be used to measure success So, our approach is to look at the different evaluation needs that are related with the different phases of a program, and linking thoses phases from an evaluation perspectie. That is, seen it as a continuos.

A framework built on existing research Comprehensive evaluation as a process Each type of evaluation is an input for the next phase of evaluation (and of the program) i.e. evidence from an evaluation phase is needed for the next phase

A definition for comprehensive evaluation A dynamic, interactive and progressive process that aims to analyze program performance using synergically a set of analytical tools that allow to identify areas of improvement considering formulation, planning and operation of programs, and providing recommendations oriented to improve programs It includes diagnosis, design, processes, implementation, results, effectiveness, efficiency

Validation of identified problem and it´s causal chain Comprehensive evaluation framework Características de la evaluación integral Validation of identified problem and it´s causal chain Analysis of diagnosis

Comprehensive evaluation framework Características de la evaluación integral Consistency between design and causal chain, target population definition Design evaluation Analysis of diagnosis

Comprehensive evaluation framework Características de la evaluación integral Verification of resources allocation Evaluation of targeting and coverage Design evaluation Analysis of diagnosis

Comprehensive evaluation framework Analysis of resources and information flows for program initiation Características de la evaluación integral Evaluation of implementation Evaluation of targeting and coverage Design evaluation Analysis of diagnosis

Comprehensive evaluation framework Validation of operation according to planning Evaluation of processes Características de la evaluación integral Evaluation of implementation Evaluation of targeting and coverage Design evaluation Analysis of diagnosis

Comprehensive evaluation framework Goals and performance analysis Evaluation of results Evaluation of processes Características de la evaluación integral Evaluation of implementation Evaluation of targeting and coverage Design evaluation Analysis of diagnosis

Comprehensive evaluation framework Measurement of attributable effect Evaluation of effectiveness Evaluation of results Evaluation of processes Características de la evaluación integral Evaluation of implementation Evaluation of targeting and coverage Design evaluation Analysis of diagnosis

Comprehensive evaluation framework Measurement of efficiency (allocation, technical, administrative) Comprehensive evaluation framework Evaluation of efficiency Evaluation of effectiveness Evaluation of results Evaluation of processes Características de la evaluación integral Evaluation of implementation Evaluation of targeting and coverage Design evaluation Analysis of diagnosis

Comprehensive evaluation framework Evaluation of efficiency Evaluation of effectiveness Evaluation of results Evaluation of processes Características de la evaluación integral Evaluation of implementation Evaluation of targeting and coverage Design evaluation Analysis of diagnosis

Measuring and accountability Evidence based design Implementation according to planning Measuring results THIS IS THE PURPOSE OF COMPREHENSIVE EVALUATION

How to state evaluation questions? Understanding the evaluatee What it is for? What is the theory behind? What is it trying to accomplish? Understanding evaluatee needs In what phase is the evaluatee? What evidence will help the evaluatee to improve its performance?

This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement AID-OAA-L-14-00004. MEASURE Evaluation is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill in partnership with ICF International; John Snow, Inc.; Management Sciences for Health; Palladium; and Tulane University. Views expressed are not necessarily those of USAID or the United States government. www.measureevaluation.org