1. The political side of social program evaluation
Ministry of Social Development
MEXICO
Gonzalo Hernández Licona

2. Objective
Analyse the institutional challenges facing Mexico, specifically SEDESOL, in constructing a Monitoring and Evaluation (M&E) system.
How can we institutionalize an M&E system?
The political constraints and challenges
How can we go beyond the impact evaluation of Oportunidades?:
More programs evaluated in a regular basis
Constructing a Results-based management system
Insert M&E within the Social Policy process

3. Outline Analytical framework What have we done and what’s next?
What type of M&E system we’re aiming for?
The need of creating and designing institutions in order to build an M&E system
What have we done and what’s next?
Conclusions

4. Social Policy Identification of social problems and objectives
Analysis
Program design
Program operation and resources
Budget
Monitoring and Evaluation

5. Evaluation: why and how
Helps re-designing and improving programs
Supports efficient use of public funds
Adds objective and technical elements to the social policy debate
Promotes Transparency and social accountability
Emphasis on results
We need to measure indicators …but consider qualitative evaluations
(Long run) impact evaluation and (frequent) monitoring of every day operation
External (and good) evaluators
Objective and useful evaluations: search for program’s stakeholders cooperation and participation

6. Decision taking Firm Social Program Process
Well-being
poverty health infrastructure
income nutrition perception
education social cap satisfaction
I. Identifying benefits
II. Measuring Impact
III. Information
IV. Who should evaluate?
V. Monitoring
VI. Who demands evaluations?
VII. What do we do with the results?
Profits
What would have happened without.. .?
Counter-factual
I-C. IRR
In general precise
The firm pays for it
Lots of information
Who pays for it?
The firm itself
External auditors
The program
External evaluators
In order to know the
whole process
It doesn’t work usually looking for results
Not clear
Public resources
Owners; share-holders
Efficient use of information
Results: what for?

7. I. Identifying benefits
Precise Rules of Operation
Better rules now 
Still problems due to the fear for auditors 
We still have social programs that have only political objectives: Opciones productivas, PET, Acuerdos para el Campo: Vivienda rural, adultos mayores del campo 

8. II. Measuring Impact: How?
Guidelines for the annual Evaluation of Programs issued in 2002 by the Ministry of Finance and the Ministry of Audit (Contraloría)
The guideline indicates the need to measure impact in every program every year
Very ambitious... 
…but helpful in the short run to accelerate the creation of a culture of evaluation 
Pressure from donors 
Progresa
CIMO
Probecat..

9. II. Measuring Impact: Promoting long-run impact evaluations
External support: technical and financial (WB, IDB, Conacyt, international academics) 
At least 9 impact evaluations in SEDESOL 
Progresa-Oportunidades
Liconsa fortified milk
Microsimulation: Oportunidades, Liconsa, Diconsa
Food program
Micro-regions Strategy
Habitat
Housing program Tu Casa
Coahuila State’s Piso Firme
Jóvenes con Oportunidades

10. II. Measuring Impact: The politics of the evaluation design
Progresa: Centralized program with relatively little participation from beneficiaries or local authorities: it favoured randomization.
Decentralized programs sometimes should seek for other methodologies
An experimental design requires political support
We tried to include in the SDL the possibility to have randomization, when feasible. 

11. III. Information
From the institutional point of view it is not clear who should pay for the information 
The program?
Sedesol?
Hacienda?
Donors?
Sometimes programs hide information from evaluators 
Next step: La Contraloría 

12. IV. Who should evaluate? Presupuesto de Egresos de la Federación
External Evaluators 
National evaluators
International evaluators not allowed sometimes 
Creation in 2002 of the Under Secretariat of Planning, Prospective and Evaluation 
Social Development Law 
External evaluators 
The SDL explicitly bans consulting firms from evaluating social programs 

13. IV. Who should evaluate?: The Human capital of External evaluators
Not enough (good) evaluators 
Impact evaluation
Monitoring
A good evaluator should be able to evaluate a program with the existing ingredients: experiments are rare.
Impact vs Monitoring 
Seminars (impact evaluation, monitoring, qualitative evaluation, power calculations, etc.) 
We’re promoting partnerships between national and international evaluators 

14. IV. Who should evaluate?: The bidding process
The bidding process favours the cheapest proposal 
Public universities don’t have to go through the bidding process 
Monopolies
Excess transparency 
Changing the external institution every year
Annual contracts
Incentives to present good results in order to evaluate again in the future 

15. V. Monitoring There is no obligation to do this 
Indicators demanded by Hacienda, Función Pública, Presidencia, Congress, with little management purpose: Highly inefficient 
International support: WB, IDB
Create a true Results-based management system for every program 
New Dirección General de Evaluación y Monitoreo de Programas Sociales, in order to promote internally the construction of a monitoring system 

16. VI. Who demands evaluations?
1. International donors 
IDB support: the need for evaluation
Receptive authorities: Levy, Gómez de León 
Internal battle (técnicos vs rudos)
2. The opposition in Congress 
Since 2000, Congress demands annual external evaluations for every public program. Presupuesto de Egresos de la Federación (PEF)
In 1999 the PRI was not a majority in Congress
The opposition feared the use of social programs for the 2000 election
There were good and reasonable academics in Government 

17. VI. Who demands evaluations?
The Social Development Law ( ), institutionalizes the evaluation process 
National Council for the Evaluation of Social Policy
Evaluation of programs not only in Sedesol
Poverty measurement
We’re including in the Social Development Law (SDL) the obligation to evaluate every new federal program 
Law of Transparency and Public Access to Information 
Democracy 

18. VII. What do we do with the results?
In the past almost nothing 
The results were useful to Progresa in order to survive, but it was not useful for the everyday management
A “small” institutional change: Our internal indicator 
% of programs evaluated every year
% of external recommendations attended by programs
The weaknesses and recommendations (summary) are officially sent to every program manager
The programs have to give an official answer on what actions will they implement
The office of internal affairs (contraloría interna), demands to see proofs of the actions taken every year

19. VII. What do we do with the results?
Evaluations are taken more into consideration
Better reports by evaluators
Programs make changes
Evaluations became this year a tool for the budget process within Sedesol 
We still need to link more closely (and formally, evaluations with the budget process) 

20. VIII. The politics of the evaluation process: very important
If we aim for an objective but yet useful M&E, we need to take into account stakeholders
We need the participation of stakeholders involved with evaluations
The DGEM published internal rules for the evaluation process: 
In the process operators should participate in TORs, analysis and reviewing sessions of developed work
Continuous dialogue with external agents
DGEM runs the party

21. Decision taking Marks Social Program Process
Well-being
poverty health infrastructure
income nutrition perception
education social cap satisfaction
I. Identifying benefits
II. Measuring Impact
III. Information
IV. Who should evaluate?
V. Monitoring
VI. Who demands evaluations?
VII. What do we do with the results? ? ?
What would have happened without.. .?
Counter-factual 
Lots of information
Who pays for it? 
The program
External evaluators  ?
It doesn’t work usually looking for results 
Not clear
Public resources   ?
Results: what for? 

22. Conclusions The Evaluation must be part of the Social Policy process
Building a Monitoring and Evaluation System is a political task, that requires technical elements
It is important to institutionalize the process and take into consideration the program’s stakeholders for the evaluation process
In Mexico, it is crucial to build-up on the evaluators, policy-makers and congress official’s technical abilities

23. Too many programs, not many evaluations
Between 1990 and 2002 Mexico spent almost 550 billion dollars in social policy
There were very few evaluations: Probecat, Liconsa, Diconsa, Progresa
The market itself doesn’t solve the need for evaluation in social programs