1. INTRODUCTION TO Monitoring and Evaluation systems Poverty reduction and equity

2. Why M&E Systems? Improve the effectiveness of public policy
Improve the allocation of public expenditure for higher welfare
Increase accountability
Strengthen design, implementation, and results of programs and policies

3. Outline of the Presentation
What is M&E?
How M&E influences Policy?
Country M&E Systems

4. 1.- What is M&E

5. Monitoring Provides regular information on performance
A continuing function
1) Indication of extent of progress towards goals
- poverty rates
- coverage of basic services
2) systematic data collection
3) usually conducted in-house

6. What to Monitor?
Identify few indicators that can be measured and are of interest to policy makers
Prioritize input, output, outcome and impact indicators for monitoring
Develop a data collection system to sustain monitoring
Inputs
Outputs
Outcomes
Impact

7. How to Monitor?
Use different types of information: surveys and censuses, administrative data, and participatory exercises
Draw on logical framework and arrangements for results monitoring which identifies:
-Data sources for each indicator
-Frequency of measurements
-The organization responsible for collecting information.
Different types of Monitoring:
-Annual progress report
-Database for continuous monitoring
-Desk reviews

8. Evaluation
Evaluation helps to analyze deviations from targets and goals
( e.g Why learning achievement increased? Why poverty decreased?)
Usually conducted by independent evaluators
A systematic and objective measurement of the results achieved by a project/program/policy

9. What to Evaluate? Evaluation can be time and resource intensive
Plan ahead evaluation and identify questions
Strategy, operation, and learning
Reserve impact evaluation in cases such as
Strategic significance of a policy or program
Contributing to close knowledge gap
Innovative nature of the policy or program

10. Trade Offs in Evaluation
Experimental
with random
assignment
High
Quasi-
experimental
with statistical
controls
Strength of
causal inference
Participant
judgment and
expert opinion
Low
Before-
and-after
Low
High
Complexity and sometimes cost

11. Summing up…
Monitoring assesses progress in implementation of ongoing programs
Evaluation provides a snapshot against some benchmarks
Monitoring looks at progress relative to targets and assumes there is causality
Evaluation seeks to establish and prove causality

12. How M&E influences Policy?

13. M&E and the Policy Cycle
M&E needs to accompany the policy cycle so decisions can be based on evidence
Define what you want  set goals and targets
Analyze what might work to reach targets  design programs and evaluate ex-ante
Implement  set performance indicators, a monitoring system , and implement evaluations

14. But we are far from there…
4 years ago, more than half of programs in Mexico had no logical frameworks
No explicit objectives, no definition of target population
Countries seldom set targets using scientific methods:
Reduce violent crimes from x to y in 5 yrs… is that a meager performance or a great performance?
Countries rarely define the outcome (say, nutrition), and then decide in which sector the correct intervention is
Countries seldom analyze ex ante who the winners and losers of a policy are

15. Definition of Objectives and Targets
The Results Chain
Definition of Objectives and Targets
Policy and Program Design
Inputs
Activities
Outputs
Outcomes
Impacts
Implementation
Results
Amount of services provided
Health literacy
Consumption, Life expectancy, poverty

16. Definition of Objectives and Targets
The Results Chain
Planning Stage
Definition of Objectives and Targets
Benchmarking
Logic Framework
Poverty Analysis
Ex-Ante Distributional and Poverty analysis
Policy and Program Design
Inputs
Activities
Outputs
Outcomes
Impacts
Implementation
Results
What are the extent and causes of poverty in a given country?
How is a proposed policy expected to affect the poor?
How to set strategic targets for a given policy/program?

17. The Results Chain (cont)
Definition of Objectives and Targets
Logical Framework
Performance Indicators
Dashboards
Monitoring Stage
Policy and Program Design
Inputs
Activities
Outputs
Outcomes
Impacts
Implementation
Results
What is a given policy/program trying to achieve?
What are the causal links through which the project is supposed to achieve its goals?
Is the project achieving its objectives? How to measure outputs and outcomes?

18. Definition of Objectives and Targets
The Results Chain
Is the execution of the project activities conducive to the expected results?
Are the chosen indicators good indicators?
What is the impact of the intervention?
Are the changes in well-being indeed attributable to the intervention?
Definition of Objectives and Targets
Policy and Program Design
Inputs
Activities
Outputs
Outcomes
Impacts
Implementation
Results
Design Evaluation
Process Evaluation
Consistency Evaluation
Impact Evaluation
Economic Analysis/Modelling
Evaluation Stage

19. Several methods to support evidence policy
Performance indicators The logical framework (logframe) approach Theory-based evaluation Formal surveys Rapid appraisal methods Participatory methods Public expenditure tracking surveys Impact evaluation Cost-benefit and cost-effectiveness analysis
EXAMPLES OF THESE TOOLS

20. Example: Consistency and Results
Consistency and Results covers:
Design, Strategic Planning, Target Population and Coverage, Operation, Perception of Beneficiaries and Final Results.
Instrument applied by external institutions under terms of reference prepared by a Central Unit
The evaluation includes a Summary
Strengths, Challenges and Recommendations for every program

21. Consistency and Results: Questions
DESIGN
Does de program identifies clearly the problem it’s trying to solve?
Is it clear that the products or services produced by the program logically contribute to reduce the problem the program is tacking?
STRATEGIC PLANNING
Does the program have updated strategic plans for the short, middle and long-term?
In the S-plan, does the final results are clearly established?
TARGET POPULATION AND COVERAGE
Does the program have a method to identify and quantify the potential and target population?
Does the program have a coverage strategy for the short, middle and long- term?

22. Consistency and Results Questions (cont)
OPERATION
Are there standard and adequate procedures for the selection of projects, beneficiaries?
Are there documents showing that the procedures are done according to the rules of operation.
PERCEPTION OF BENEFICIARIES
Does the program have instruments to measure the satisfaction of the beneficiaries with the program?
FINAL RESULTS
Does the program systematically collect objective information about its Goal (Fin y Propósito)?
Does the program have rigorous impact evaluations?

23. Findings CONEVAL-Mexico
Topic
Findings
DESIGN
67%
have a Purpose and Goal that relate to the problem’s solution.
50%
of the programs defined necessary and enough outputs to achieve the Purpose.
STRATEGIC PLANNING
21%
of the plans clearly establish the results they want to accomplish.
23%
of the programs have strategic plans in the short, middle and long-term.
TARGET POPULATION AND COVERAGE
24%
of the programs quantify the target and potential population.
of the programs have a coverage strategy for the short, middle and long-term.

24. Findings CONEVAL-Mexico
Topic
Findings
OPERATION
72%
have standardized and adequate procedures for the project/ beneficiaries selection.
84%
of the programs have efficacy indicators in their operation.
BENEFICIARIES PERCEPTION
50%
of the programs have instruments to measure the satisfaction level of the target population.
FINAL RESULTS
34%
of the programs collect objective information about their final results.
26%
have external evaluations that allow to measure their impact.

25. Country M&E Systems

26. Country M&E Systems: Keys for success
Systems which permit the generation of information (basic data, indicators, evaluations, etc), reliable, credible and high quality
A high level of utilization of the information generated by the M&E system
Sustainability for the future
Improve the supply
Incentives from the demand side

27. The demand side: specific uses of an M&E system (or..why it’s needed?)
To support government planning at the national, sub national, local and sectoral level in the establishment of targets/goals and permanently ask the question why are targets not being met?  focus on results
To support management public programs through monitor and evaluate performance and effectiveness
To informing the redesign and design of programs and interventions
To inform budget allocation decisions
To facilitate accountability
Countries choose one or all of these roles for M&E systems
Define strategic goals and quantitative targets
Define programs consistent with the targets
Monitor and evaluate those programs: er you doing thigns right? And are you doing the right things? Element of any results based management system
Contradictoy?

28. Increasing effective demand and use of M&E
Pressure from civil society for budget accountability and vigilance of state´s performance
Need to maintain macro - equilibrium and simultaneous pressures to extend coverage of public services  efficiency
Produce information (data , indicators, evaluations) which can effectively be used. Worry about the trust on the quality of information
Incentives for sectoral and budget officials, congress, etc. so that they use the information
Leaders who persuade and motivate the government to use the information generated  a champion (person or institution) of results-orientation
All this is usually more important than legislation

29. Building blocks on the supply side
1. Improve Statistics:
household surveys, sectoral surveys and censuses
Better administrative records and sectoral statistics
Integration of systems to generate input and outcome indicators
Regional and local level data
Albania: The main purpose of the sub-national reports was to explore and demonstrate how the MDGs could be used to build support and momentum for the MDGs from the bottom up, to assess local poverty challenges and to serve as an engine for comprehensive local development. Along with the preparation of sub-national development
strategies, the reports formed an important part of activities initiated by UNDP and other members of the UN
system in response to the first national MDG report
Brazil: The campaign sought to raise nation-wide awareness because ultimate success depends on action by many and diverse people. A single overarching message was chosen to unite the campaign. It is simple, yet empowering, resonates with many audiences and combines education with a call to action: ‘8 ways to change the world. Yes we can.’ The campaign became known by its Portuguese tagline, Nós Podemos. Making the Initial Policy Link to the MDGs
Ethiopia: An initial turning point for the MDGs in Ethiopia can be traced to the Consultative Group Meeting in December 2002, when the Government presented the PRSP to donors for funding. For the first time, the Government articulated its commitment to base the PRSP on the MDGs. The meeting’s communiqué referred to the PRSP “as the primary development programme for Ethiopia,providing a shared platform to achieve accelerated growth to overcome the country’s pervasive poverty and food insecurity and for reaching the Millennium Development Goals.” The MDG agenda started moving more significantly thereafter, in 2003

30. to feed the production of M&E
Sectoral/program monitoring systems
Government –wide monitoring systems (Systems of presidential goals, PRS monitoring).
Performance evaluation systems for public services
Ex-ante and Ex-post Evaluations (Process evaluations , Rapid Assessments, Impact E.)

31. Ensuring sustainability
Permanently redefine the system. M&E tools have to be effectively used to improve performance and effectiveness of the state at all levels:
Are we spending right?
Are programs being well implemented?
Are programs attaining its targets?
Are we spending on the right things?
Are programs being effective
Establishing strategic alliances: M&E a permanent function of different entities that should work together
Finance ministries
Planning agencies,
Supreme Audit Institutions
National Statistical Offices
Evaluation offices ( embedded in an high leverage institutions)
Sector ministries and its statistical offices
Local governments

32. Thanks you