IMPACT EVALUATION FOR POLICY MAKING A look at LAC countries with weaker research capacities María Laura Alzúa Habiba Djebbari Martín Valdivia Session: Demand and Supply of Impact Evaluations Mind the Gap: From Evidence to Policy Impact Cuernavaca, June 15-17, 2011

Outline Why do we need to evaluate the impact of programs? The Evidence to Policy Process Key Hypotheses of this study What we do? Some preliminary results

The need for impact evaluations Money for foreign aid/taxpayers have alternative uses for a set of goals Social behavior and economic policies – Modeling incentives, behavioral constraints and the objectives of policies/programs (Karlan & Appel, 2011; Banerjee & Dufló, 2011) – Hidden/unexpected effects The challenge to identify a sound counterfactual – What would have happened to the beneficiaries in the absence of the program? Experimental methods Non-experimental methods (PSM, IV, RDD, etc.)

Research questions How many studies have been done and on what topics or policy issues and with what methodologies (RCT, q- experimental, IV, etc.)? What are the emerging fields or sectors of interested being evaluated? Who is doing it? The role of local universities and research centers vis-à-vis the work by northern-based researchers and institutions and initiatives such as J-PAL, IPA, and donors and the multilaterals, etc. Who is funding it (3IE, Gates, MCC, the multilateral banks)? How have their project approval processes being modified to give stellar importance to impact evaluation designs? How are they being used to shape policy? What are the institutional arrangements that can better insert impact evaluations into policy design?

Key hypotheses The number of programs with rigorous IEs have increased significantly in the region, especially in countries with more capacities for research and in policy making – Institutionalization of the use of IEs for policy making has been key

What we do? Collect all IE studies done in the region since 1995, and especially in the following set of countries – South America: Bolivia, Ecuador, Paraguay, Perú – Central America: El Salvador, Guatemala, Honduras, Nicaragua – Caribbean: Dominican Republic, Jamaica, Trinidad and Tobago, Guyana Sources: – Bouillón & Tejerina (2007) – Most common academic papers databases: IDEAS/RePEC, EconLit and JSTOR, and SSRN Randomized Social Experiments – Other databases: World Bank DIME, IADB OVE/Development Effectiveness, IPA, J-PAL, 3ie, MCA, the Bill and Melinda Gates Foundation, IFPRI

What we do? Final selection criteria restricts the sample to those with reasonable identification strategies – We exclude studies with identification strategies based on community characteristics as IVs, PSM without baseline, etc. Three case studies: Dominican Republic, El Salvador, Peru – What are the restrictions to the implementation and use of rigorous IEs? – Are there conditions for the institutionalization of IEs? (Briceño and Gaarder, 2009) Supply of training courses on IE in LAC – Sources: IPA, J-PAL, World Bank, IADB, IEN-LACEA – Who are they serving? Local policy makers, researchers