1. CONSISTENCY AND RESULTS EVALUATION IN MEXICO SEMPTEMBER 2008 Gonzalo Hernández Licona

2. The Context: Performance Evaluation System (SED) Consistency and Results Evaluation

3. Results-Based M&E System Strategic goals: Ministries Program indicators (Results): Logical Framework Results Follow up on evaluation results Consistency and Results Evaluation Impact Evaluation Process Evaluation Performance Evaluation Strategic Evaluation Annual Evaluation Program Planning and IndicatorsEvaluation Programs aligned to Strategic goals

4. Performance Evaluation System (SED). Systematic information about the program’s performance I I I I I E E I I Beginning in 2009:

5. % Budget Spent 0 20 40 60 80 100 1er trimester2o trimester3er trimester4to trimester 2009 (Scholarship Program) SED. Indicators 1

6. Number of scholarships granted 0 50,000 100,000 150,000 200,000 250,000 1er trimester2o trimester3er trimester4to trimester 2009 Goal (Scholarship Program) SED. Indicators 2

7. SED. Indicators 3 Coverage Indicator 0 10 20 30 40 50 60 70 80 90 100 200620082010 201220142016201820202022 2024 (Scholarship Program)

8. Impact in educational years 0 6 7 8 9 10 11 12 13 14 15 20082009 2010 201120122013201420152016 2017 (Scholarship Program) Beneficiaries Control Group SED. Evaluation 1

9. Consistency and Results Evaluation. What is it? The main goal: to what extent social programs are oriented toward results, based on their own documentation and processes? It’s a diagnosis about the institutional and management capacity of the programs to achieve results. The idea was to have a homogeneous instrument, with measurable answers, in order to compare performance among programs. The evaluation took place in 2007-2008 for 119 programs. This evaluation is a baseline showing what type of tools (programs) the new Government has to achieve its goals. The next evaluation will take place in 3 years time.

10. Consistency and Results Evaluation. Characteristics The evaluation has 100 questions about the following topics, for every program: Design, Strategic Planning, Target Population and Coverage, Operation, Perception of Beneficiaries and Final Results. The Design topic is mainly based on the Logical Framework. 80 of the questions have a binary answer (YES/ NO) The evaluations were done by external institutions (universities, consultants, experts), hired by the ministries. 7 of them hired by CONEVAL. The evaluation includes a Summary:Strengths, Challenges and Recommendations for every program.

11. DESIGN  Does de program identifies clearly the problem it’s trying to solve?  Is it clear that the products or services produced by the program logically contribute to reduce the problem the program is tacking? STRATEGIC PLANNING  Does the program have updated strategic plans for the short, middle and long-term?  In the S-plan, does the final results are clearly established? TARGET POPULATION AND COVERAGE  Does the program have a method to identify and quantify the potential and target population?  Does the program have a coverage strategy for the short, middle and long-term? Consistency and Results Evaluation. Questions

12. OPERATION  Are there standard and adequate procedures for the selection of projects, beneficiaries?  Are there documents showing that the procedures are done according to the rules of operation. PERCEPTION OF BENEFICIARIES  Does the program have instruments to measure the satisfaction of the beneficiaries with the program? FINAL RESULTS  Does the program systematically collect objective information about its Goal (Fin y Propósito)?  Does the program have rigorous impact evaluations? Consistency and Results Evaluation. Questions

13. Consistency and Results Evaluation. General Findings TopicFindings DESIGN 67% have a Purpose and Goal that belong to the problem’s solution. 50% of the programs defined necessary and enough outputs to achieve the Purpose. STRATEGIC PLANNING 21% of the plans clearly establish the results that want to accomplish. 23% of the programs have strategic plans in the short, middle and long-term. TARGET POPULATION AND COVERAGE 24% of the programs quantify the target and potential population. 23% of the programs have a coverage strategy for the short, middle and long-term.

14. Consistency and Results Evaluation. General Findings TopicFindings OPERATION 72% have standard and adequate procedures for the project/ beneficiaries selection. 84% of the programs have efficacy indicators in their operation. BENEFICIARIES PERCEPTION 50% of the programs have instruments to measure the satisfaction level of the target population. FINAL RESULTS 34% of the programs collect objective information about their final results. 26% have external evaluations that allow to measure their impact.

15. Consistency and Results Evaluation. Findings per Program

16. GOAL 5 : Reduce extreme poverty and assure equal opportunities. CORE: Equal Opportunities THEME: Poverty Alleviation Milk Supply Program (LICONSA) GOAL Improve the nutritional level of poor children and pregnant women. PURPOSE Beneficiaries have access to fortified milk. EVALUATION / RESULTS TopicDesign Strategic Planning Target Population and Coverage Operation Final Results Results are being documented Type of Results % 100%71%50%100% Score

17. TOPICANSWERQUESTION DESIGN  The Outputs allow the program to achieve its Purpose  The program Purpose is unique and reflex a significant benefit in the attended population  The Purpose contributes directly to accomplish the program Goal  The program Goal is linked with a Strategic Goal of the Ministry STRATEGIC PLANNING  The plans establish the results that the program seek  The program have enough indicators that reflex it Purpose  The program have pertinent goals and specific deadlines for its performance indicators  The evaluations results are used to improve the performance of the program  There are updated strategic plans in the short, middle and long-term  The strategic plans establish indicators, goals, politics strategies and work schedules  There are adequate mechanisms to define goals and indicators TARGET POPULATION AND COVERAGE  The program has a method to quantify and determinate the potential and target population  The coverage advance is adequate to accomplish the program Goal and Purpose  The program has a coverage strategy in the short, middle and long-term  The coverage strategy is adequate OPERATION  The support granted complied the established regulations  There is documentary evidence that the program follow the implementation processes  There are adequate standardize processes for the project/ beneficiaries selection  There are adequate procedures to receive and process support applications  There is systematized data which allows the timely monitoring of the benefits execution  The program has an organizational structure to produce the Outputs and achieve the program Purpose  The program has effectiveness indicators of the operation  There is a adequate systematization in the program management and operation FINAL RESULTS Impact Evaluations Results Results of the Program Beneficiaries Monitor Other Notable Results The program do not document any Final Results Adequate Final Results Enough Final Results Insufficient Final Results Final Results unknown Milk Supply Program (LICONSA) GOAL 5 : To reduce the extreme poverty and to assure equal opportunities. CORE: Equal Opportunities THEME: Poverty Alleviation

18. Milk Supply Program (LICONSA) GOAL 5 : To reduce the extreme poverty and to assure equal opportunities. CORE: Equal Opportunities THEME: Poverty Alleviation

19. Milk Supply Program (LICONSA) GOAL 5 : To reduce the extreme poverty and to assure equal opportunities. CORE: Equal Opportunities THEME: Poverty Alleviation

20. Rural Housing Program GOAL To contribute to the quality of life of indigenous and rural families in patrimony poverty that live in localities of high and very high marginalization, through housing solutions PURPOSE Mexican families living in rural and indigenous localities with high and very high levels of marginalization improve their housing conditions EVALUATION / RESULTS TopicDesign Strategic Planning Target Population and Coverage Operation Resultados Finales Document Results Type of Results % 44%14%0%38% Score GOAL 5 : Reduce extreme poverty and assure equal opportunities. CORE: Equal Opportunities THEME: Poverty Alleviation

21. Consistency and Results Evaluation. What did we do? EVALUATION: ACCOUNTABILITY AND PERFORMANCE IMPROVEMENT  The evaluations were sent to Congress and the Ministry of Finance in April  The complete evaluations can be found in the web page (CONEVAL and Ministries).  The scores were sent to Presidency in order to take better budget decisions for 2009. We don’t know the results yet…  Nowadays, programs are analyzing the pertinent and feasible recommendations given in the Consistency and Results Evaluation, and they are doing a Work Plan to improve their performance.  This Work Plan will be public.

22. Summary The ECR is part of the SED. It’s perhaps one of the first instruments of the SED. We were able to get a homogeneous picture of 119 programs. We’re improving the instrument. The information will be used to improve the program The information is in the web. The President wanted the information for budget purposes, but we don’t know if this was the case. The Work Plan to improve the program, will be a bridge connecting Accountability and Performance improving.

23. 2009

24. SED. Valuable information menu I I I I I E E I I Performance Evaluation (Annual Summary)

25. Annual Performance Evaluation 4 pages report, summarizing the information contained in the SED (to be read by Congress): The Report will have:  Name and goal  Budget (last 3 years)  Results (shown in SED; through indicators or Evaluations)  Coverage (shown in SED)  Financial indicators (shown in SED)  ….  Recommendations (1 or 2)  If there is no information about some of these subjects, it must be reported

26. Consistency and Results Evaluation. Challenges Huge variance in the quality of external evaluators.  CONEVAL re-evaluated 47 evaluations. Questions:  A lot of questions; some duplications.  We are improving the instrument. The evaluation was highly linked to the Logical Framework.  The programs (and evaluators) had just started to know this methodology.