INTRODUCTION TO Monitoring and Evaluation systems Poverty reduction and equity gacevedo@worldbank.org

Why M&E Systems? Improve the effectiveness of public policy Improve the allocation of public expenditure for higher welfare Increase accountability Strengthen design, implementation, and results of programs and policies

Outline of the Presentation What is M&E? How M&E influences Policy? Country M&E Systems

1.- What is M&E

Monitoring Provides regular information on performance A continuing function 1) Indication of extent of progress towards goals - poverty rates - coverage of basic services 2) systematic data collection 3) usually conducted in-house

What to Monitor? Identify few indicators that can be measured and are of interest to policy makers Prioritize input, output, outcome and impact indicators for monitoring Develop a data collection system to sustain monitoring Inputs Outputs Outcomes Impact

How to Monitor? Use different types of information: surveys and censuses, administrative data, and participatory exercises Draw on logical framework and arrangements for results monitoring which identifies: -Data sources for each indicator -Frequency of measurements -The organization responsible for collecting information. Different types of Monitoring: -Annual progress report -Database for continuous monitoring -Desk reviews

Evaluation Evaluation helps to analyze deviations from targets and goals ( e.g Why learning achievement increased? Why poverty decreased?) Usually conducted by independent evaluators A systematic and objective measurement of the results achieved by a project/program/policy

What to Evaluate? Evaluation can be time and resource intensive Plan ahead evaluation and identify questions Strategy, operation, and learning Reserve impact evaluation in cases such as Strategic significance of a policy or program Contributing to close knowledge gap Innovative nature of the policy or program

Trade Offs in Evaluation Experimental with random assignment High Quasi- experimental with statistical controls Strength of causal inference Participant judgment and expert opinion Low Before- and-after Low High Complexity and sometimes cost

Summing up… Monitoring assesses progress in implementation of ongoing programs Evaluation provides a snapshot against some benchmarks Monitoring looks at progress relative to targets and assumes there is causality Evaluation seeks to establish and prove causality

How M&E influences Policy?

M&E and the Policy Cycle M&E needs to accompany the policy cycle so decisions can be based on evidence Define what you want  set goals and targets Analyze what might work to reach targets  design programs and evaluate ex-ante Implement  set performance indicators, a monitoring system , and implement evaluations

But we are far from there… 4 years ago, more than half of programs in Mexico had no logical frameworks No explicit objectives, no definition of target population Countries seldom set targets using scientific methods: Reduce violent crimes from x to y in 5 yrs… is that a meager performance or a great performance? Countries rarely define the outcome (say, nutrition), and then decide in which sector the correct intervention is Countries seldom analyze ex ante who the winners and losers of a policy are

Definition of Objectives and Targets The Results Chain Definition of Objectives and Targets Policy and Program Design Inputs Activities Outputs Outcomes Impacts Implementation Results Amount of services provided Health literacy Consumption, Life expectancy, poverty

Definition of Objectives and Targets The Results Chain Planning Stage Definition of Objectives and Targets Benchmarking Logic Framework Poverty Analysis Ex-Ante Distributional and Poverty analysis Policy and Program Design Inputs Activities Outputs Outcomes Impacts Implementation Results What are the extent and causes of poverty in a given country? How is a proposed policy expected to affect the poor? How to set strategic targets for a given policy/program?

The Results Chain (cont) Definition of Objectives and Targets Logical Framework Performance Indicators Dashboards Monitoring Stage Policy and Program Design Inputs Activities Outputs Outcomes Impacts Implementation Results What is a given policy/program trying to achieve? What are the causal links through which the project is supposed to achieve its goals? Is the project achieving its objectives? How to measure outputs and outcomes?

Definition of Objectives and Targets The Results Chain Is the execution of the project activities conducive to the expected results? Are the chosen indicators good indicators? What is the impact of the intervention? Are the changes in well-being indeed attributable to the intervention? Definition of Objectives and Targets Policy and Program Design Inputs Activities Outputs Outcomes Impacts Implementation Results Design Evaluation Process Evaluation Consistency Evaluation Impact Evaluation Economic Analysis/Modelling Evaluation Stage

Several methods to support evidence policy Performance indicators The logical framework (logframe) approach Theory-based evaluation Formal surveys Rapid appraisal methods Participatory methods Public expenditure tracking surveys Impact evaluation Cost-benefit and cost-effectiveness analysis EXAMPLES OF THESE TOOLS

Example: Consistency and Results Consistency and Results covers: Design, Strategic Planning, Target Population and Coverage, Operation, Perception of Beneficiaries and Final Results. Instrument applied by external institutions under terms of reference prepared by a Central Unit The evaluation includes a Summary Strengths, Challenges and Recommendations for every program

Consistency and Results: Questions DESIGN Does de program identifies clearly the problem it’s trying to solve? Is it clear that the products or services produced by the program logically contribute to reduce the problem the program is tacking? STRATEGIC PLANNING Does the program have updated strategic plans for the short, middle and long-term? In the S-plan, does the final results are clearly established? TARGET POPULATION AND COVERAGE Does the program have a method to identify and quantify the potential and target population? Does the program have a coverage strategy for the short, middle and long- term?

Consistency and Results Questions (cont) OPERATION Are there standard and adequate procedures for the selection of projects, beneficiaries? Are there documents showing that the procedures are done according to the rules of operation. PERCEPTION OF BENEFICIARIES Does the program have instruments to measure the satisfaction of the beneficiaries with the program? FINAL RESULTS Does the program systematically collect objective information about its Goal (Fin y Propósito)? Does the program have rigorous impact evaluations?

Findings CONEVAL-Mexico Topic Findings DESIGN 67% have a Purpose and Goal that relate to the problem’s solution. 50% of the programs defined necessary and enough outputs to achieve the Purpose. STRATEGIC PLANNING 21% of the plans clearly establish the results they want to accomplish. 23% of the programs have strategic plans in the short, middle and long-term. TARGET POPULATION AND COVERAGE 24% of the programs quantify the target and potential population. of the programs have a coverage strategy for the short, middle and long-term.

Findings CONEVAL-Mexico Topic Findings OPERATION 72% have standardized and adequate procedures for the project/ beneficiaries selection. 84% of the programs have efficacy indicators in their operation. BENEFICIARIES PERCEPTION 50% of the programs have instruments to measure the satisfaction level of the target population. FINAL RESULTS 34% of the programs collect objective information about their final results. 26% have external evaluations that allow to measure their impact.

Country M&E Systems

Country M&E Systems: Keys for success Systems which permit the generation of information (basic data, indicators, evaluations, etc), reliable, credible and high quality A high level of utilization of the information generated by the M&E system Sustainability for the future Improve the supply Incentives from the demand side

The demand side: specific uses of an M&E system (or..why it’s needed?) To support government planning at the national, sub national, local and sectoral level in the establishment of targets/goals and permanently ask the question why are targets not being met?  focus on results To support management public programs through monitor and evaluate performance and effectiveness To informing the redesign and design of programs and interventions To inform budget allocation decisions To facilitate accountability Countries choose one or all of these roles for M&E systems Define strategic goals and quantitative targets Define programs consistent with the targets Monitor and evaluate those programs: er you doing thigns right? And are you doing the right things? Element of any results based management system Contradictoy?

Increasing effective demand and use of M&E Pressure from civil society for budget accountability and vigilance of state´s performance Need to maintain macro - equilibrium and simultaneous pressures to extend coverage of public services  efficiency Produce information (data , indicators, evaluations) which can effectively be used. Worry about the trust on the quality of information Incentives for sectoral and budget officials, congress, etc. so that they use the information Leaders who persuade and motivate the government to use the information generated  a champion (person or institution) of results-orientation All this is usually more important than legislation

Building blocks on the supply side 1. Improve Statistics: household surveys, sectoral surveys and censuses Better administrative records and sectoral statistics Integration of systems to generate input and outcome indicators Regional and local level data Albania: The main purpose of the sub-national reports was to explore and demonstrate how the MDGs could be used to build support and momentum for the MDGs from the bottom up, to assess local poverty challenges and to serve as an engine for comprehensive local development. Along with the preparation of sub-national development strategies, the reports formed an important part of activities initiated by UNDP and other members of the UN system in response to the first national MDG report Brazil: The campaign sought to raise nation-wide awareness because ultimate success depends on action by many and diverse people. A single overarching message was chosen to unite the campaign. It is simple, yet empowering, resonates with many audiences and combines education with a call to action: ‘8 ways to change the world. Yes we can.’ The campaign became known by its Portuguese tagline, Nós Podemos. Making the Initial Policy Link to the MDGs Ethiopia: An initial turning point for the MDGs in Ethiopia can be traced to the Consultative Group Meeting in December 2002, when the Government presented the PRSP to donors for funding. For the first time, the Government articulated its commitment to base the PRSP on the MDGs. The meeting’s communiqué referred to the PRSP “as the primary development programme for Ethiopia,providing a shared platform to achieve accelerated growth to overcome the country’s pervasive poverty and food insecurity and for reaching the Millennium Development Goals.” The MDG agenda started moving more significantly thereafter, in 2003

to feed the production of M&E Sectoral/program monitoring systems Government –wide monitoring systems (Systems of presidential goals, PRS monitoring). Performance evaluation systems for public services Ex-ante and Ex-post Evaluations (Process evaluations , Rapid Assessments, Impact E.)

Ensuring sustainability Permanently redefine the system. M&E tools have to be effectively used to improve performance and effectiveness of the state at all levels: Are we spending right? Are programs being well implemented? Are programs attaining its targets? Are we spending on the right things? Are programs being effective Establishing strategic alliances: M&E a permanent function of different entities that should work together Finance ministries Planning agencies, Supreme Audit Institutions National Statistical Offices Evaluation offices ( embedded in an high leverage institutions) Sector ministries and its statistical offices Local governments

Thanks you