Impact Evaluation in Mexico Harry Anthony Patrinos HDNED

Country Context for Evaluation Capacity in M&E and evaluation precedents Good education data/information Lack of openness on information in education Lack of capacity in education for IE

Progresa/Oportunidades Large scale CCT program Randomized design and phased roll out External and internal capacity for IE Results published and used for programming Led to evaluation institutions, capacity throughout, legal framework Pressure on other sectors

Education Impact Evaluations pre-2005 Isolated efforts, not large scale, few published No culture of IE No randomized experiments But interest in results and promoting good programs

Education Impact Evaluations post-2005 PAREIB – Compensatory rural educationPAREIB – Compensatory rural education PEC – School-based management in urban disadvantaged areas

Education Impact Evaluations post-2005 PAREIB – Compensatory rural education –Large rural program targeting disadvantaged rural schools, expansion to all 32 states based on evaluation in early 1990s (led by Bank) –Little interest in IE after that, but considerable capacity to collect information, monitor program –Considered successful program PEC – School-based management in urban disadvantaged areas –“Pilot” program, 2001, one of first to start with evaluation –But design deficient, not random, led to comparison problems –Considered up until last year as not successful

Education Impact Evaluations post-2005 PAREIB – Compensatory rural education –Bank undertakes simple matching study –Original resistance by authorities –Positive results led to interest and further work PEC – School-based management in urban disadvantaged areas –Bank asked to explain their lack of significant findings –Incorporates PEC in AAA and conducts IE –Shown to be positive (verified by external evaluations as well)

Pragmatic Design Used existing administrative data, project data, MoE data and surveys, Oportunidades data Initially Bank-led, then partnered with authorities and local researchers Now client carrying out, publishing, designing evaluations

Results PAREIB: Matching before project –New team focused on research, first randomization of 1 component planned PEC: Before approval, weak design –Bank study –Harvard study –Led to new design in project

Results 2 Quality of dialogue improves Project design more solid Financing project becomes sustainable

Results 3 Justified follow-on operation Lent credibility to investments, Bank role Being used for public debate

Taking Advantage of Opportunities Look at administrative data Involve operational staff Partner with program operators Know incentives of operators and client