The political side of social program evaluation Ministry of Social Development MEXICO Gonzalo Hernández Licona

Objective Analyse the institutional challenges facing Mexico, specifically SEDESOL, in constructing a Monitoring and Evaluation (M&E) system. How can we institutionalize an M&E system? The political constraints and challenges How can we go beyond the impact evaluation of Oportunidades?: More programs evaluated in a regular basis Constructing a Results-based management system Insert M&E within the Social Policy process

Outline Analytical framework What have we done and what’s next? What type of M&E system we’re aiming for? The need of creating and designing institutions in order to build an M&E system What have we done and what’s next? Conclusions

Social Policy Identification of social problems and objectives Analysis Program design Program operation and resources Budget Monitoring and Evaluation

Evaluation: why and how Helps re-designing and improving programs Supports efficient use of public funds Adds objective and technical elements to the social policy debate Promotes Transparency and social accountability Emphasis on results We need to measure indicators …but consider qualitative evaluations (Long run) impact evaluation and (frequent) monitoring of every day operation External (and good) evaluators Objective and useful evaluations: search for program’s stakeholders cooperation and participation

Decision taking Firm Social Program Process Well-being poverty health infrastructure income nutrition perception education social cap. satisfaction I. Identifying benefits II. Measuring Impact III. Information IV. Who should evaluate? V. Monitoring VI. Who demands evaluations? VII. What do we do with the results? Profits What would have happened without.. .? Counter-factual I-C. IRR In general precise The firm pays for it Lots of information Who pays for it? The firm itself External auditors The program External evaluators In order to know the whole process It doesn’t work usually looking for results Not clear Public resources Owners; share-holders Efficient use of information Results: what for?

I. Identifying benefits Precise Rules of Operation Better rules now  Still problems due to the fear for auditors  We still have social programs that have only political objectives: Opciones productivas, PET, Acuerdos para el Campo: Vivienda rural, adultos mayores del campo 

II. Measuring Impact: How? Guidelines for the annual Evaluation of Programs issued in 2002 by the Ministry of Finance and the Ministry of Audit (Contraloría) The guideline indicates the need to measure impact in every program every year Very ambitious...  …but helpful in the short run to accelerate the creation of a culture of evaluation  Pressure from donors  Progresa CIMO Probecat..

II. Measuring Impact: Promoting long-run impact evaluations External support: technical and financial (WB, IDB, Conacyt, international academics)  At least 9 impact evaluations in SEDESOL  Progresa-Oportunidades 1997-2004 Liconsa fortified milk Microsimulation: Oportunidades, Liconsa, Diconsa Food program Micro-regions Strategy Habitat Housing program Tu Casa Coahuila State’s Piso Firme Jóvenes con Oportunidades

II. Measuring Impact: The politics of the evaluation design Progresa: Centralized program with relatively little participation from beneficiaries or local authorities: it favoured randomization. Decentralized programs sometimes should seek for other methodologies An experimental design requires political support We tried to include in the SDL the possibility to have randomization, when feasible. 

III. Information From the institutional point of view it is not clear who should pay for the information  The program? Sedesol? Hacienda? Donors? Sometimes programs hide information from evaluators  Next step: La Contraloría 

IV. Who should evaluate? Presupuesto de Egresos de la Federación External Evaluators  National evaluators International evaluators not allowed sometimes  Creation in 2002 of the Under Secretariat of Planning, Prospective and Evaluation  Social Development Law  External evaluators  The SDL explicitly bans consulting firms from evaluating social programs 

IV. Who should evaluate?: The Human capital of External evaluators Not enough (good) evaluators  Impact evaluation Monitoring A good evaluator should be able to evaluate a program with the existing ingredients: experiments are rare. Impact vs Monitoring  Seminars (impact evaluation, monitoring, qualitative evaluation, power calculations, etc.)  We’re promoting partnerships between national and international evaluators 

IV. Who should evaluate?: The bidding process The bidding process favours the cheapest proposal  Public universities don’t have to go through the bidding process  Monopolies Excess transparency  Changing the external institution every year Annual contracts Incentives to present good results in order to evaluate again in the future 

V. Monitoring There is no obligation to do this  Indicators demanded by Hacienda, Función Pública, Presidencia, Congress, with little management purpose: Highly inefficient  International support: WB, IDB Create a true Results-based management system for every program  New Dirección General de Evaluación y Monitoreo de Programas Sociales, in order to promote internally the construction of a monitoring system 

VI. Who demands evaluations? 1. International donors  IDB support: the need for evaluation Receptive authorities: Levy, Gómez de León  Internal battle (técnicos vs rudos) 2. The opposition in Congress  Since 2000, Congress demands annual external evaluations for every public program. Presupuesto de Egresos de la Federación (PEF) In 1999 the PRI was not a majority in Congress The opposition feared the use of social programs for the 2000 election There were good and reasonable academics in Government 

VI. Who demands evaluations? The Social Development Law (2003-2004), institutionalizes the evaluation process  National Council for the Evaluation of Social Policy Evaluation of programs not only in Sedesol Poverty measurement We’re including in the Social Development Law (SDL) the obligation to evaluate every new federal program  Law of Transparency and Public Access to Information  Democracy 

VII. What do we do with the results? In the past almost nothing  The results were useful to Progresa in order to survive, but it was not useful for the everyday management A “small” institutional change: Our internal indicator  % of programs evaluated every year % of external recommendations attended by programs The weaknesses and recommendations (summary) are officially sent to every program manager The programs have to give an official answer on what actions will they implement The office of internal affairs (contraloría interna), demands to see proofs of the actions taken every year

VII. What do we do with the results? Evaluations are taken more into consideration Better reports by evaluators Programs make changes Evaluations became this year a tool for the budget process within Sedesol  We still need to link more closely (and formally, evaluations with the budget process) 

VIII. The politics of the evaluation process: very important If we aim for an objective but yet useful M&E, we need to take into account stakeholders We need the participation of stakeholders involved with evaluations The DGEM published internal rules for the evaluation process:  In the process operators should participate in TORs, analysis and reviewing sessions of developed work Continuous dialogue with external agents DGEM runs the party

Decision taking Marks Social Program Process Well-being poverty health infrastructure income nutrition perception education social cap. satisfaction I. Identifying benefits II. Measuring Impact III. Information IV. Who should evaluate? V. Monitoring VI. Who demands evaluations? VII. What do we do with the results? ? ? What would have happened without.. .? Counter-factual  Lots of information Who pays for it?  The program External evaluators  ? It doesn’t work usually looking for results  Not clear Public resources   ? Results: what for? 

Conclusions The Evaluation must be part of the Social Policy process Building a Monitoring and Evaluation System is a political task, that requires technical elements It is important to institutionalize the process and take into consideration the program’s stakeholders for the evaluation process In Mexico, it is crucial to build-up on the evaluators, policy-makers and congress official’s technical abilities

Too many programs, not many evaluations Between 1990 and 2002 Mexico spent almost 550 billion dollars in social policy There were very few evaluations: Probecat, Liconsa, Diconsa, Progresa The market itself doesn’t solve the need for evaluation in social programs