1. 1 Validating Ex Ante Impact Evaluation Models: An Example from Mexico Francisco H.G. Ferreira Phillippe G. Leite Emmanuel Skoufias The World Bank PREM Learning Forum-April 22, 2008

2. Introduction Conditional Cash Transfer (CCT) programs are becoming an important element of social policy in LAC Distinguishing characteristic of CCTs: social accountability supported by rigorous impact evaluation (IE)

3. Introduction Alternative IE designs: Experimental design: hh randomly assigned to T and C groups, prior to implementation of program. Typically hh surveyed in baseline and for 1 or more rounds after the start of the program. +: provide most reliable estimates (gold standard) of program impacts -: costly, likely to be of small scale -: large time lags involved

4. Introduction Quasi-experimental designs: typically comparison/control hh are obtained ex- post (after the start of the program), attempting to equalize selection bias between treatment and control groups +: less costly -: lack of baseline data (and/or pre-program differences) Overall, ex-post methods do not provide ANY information about the possible effects of the program prior to its implementation.

5. Introduction zEx ante methods: simulate the effects of the program on the basis of a structural (or reduced form) model of household behavior yEasily implemented using a representative hh data set (e.g. BFL, 2003) yExpand the set of policy-relevant questions that can be addressed, e.g. useful in designing the program, size of transfer, etc. yBased on the concept of treatment and comparison/counterfactual group yHowever, require some strong assumptions about: xFunctional form xPerfect implementation of the program xAbsence of time or trend effects.

6. Introduction z This paper is one of the first to provide a validation test of the ex-ante evaluation methodology z Approach: Use household survey data from two CCT programs (PROGRESA in Mexico and BDH-Bono de Desarollo Humano in Ecuador) where experimental designs were employed to (ex post) evaluate program impact z Use the baseline data from each survey to apply ex-ante evaluation methods to predict program impact.

7. Introduction z Compare the impact predictions obtained with the ex-ante method to the impact estimates obtained using the experimental (ex-post) methods.

8. Some Background on PROGRESA z What is PROGRESA? y Targeted cash transfer program conditioned on families visiting health centers regularly and on children attending school regularly. y Cash transfer-alleviates short-term poverty y Human capital investment-alleviates poverty in the long-term y Started in 1998. By the end of 2004: program (renamed Oportunidades) covered nearly 5 million families, in 72,000 localities in all 31 states (budget of about US$2.5 billion). y Transfers given to mothers: 20% of hh consumption expenditure

9. Some Background on PROGRESA z Two-stage Selection process: y Geographic targeting (used census data to identify poor localities) y Within Village household-level targeting (village household census) x Used hh income, assets, and demographic composition to estimate the probability of being poor (Inc per cap<Standard Food basket). x Discriminant analysis applied separately by region x Discriminant score of each household compared to a threshold value (high DS=Noneligible, low DS=Eligible) y Initially 52% eligible, then revised selection process so that 78% eligible. But many of the “new poor” households did not receive benefits

10. 10 Ex ante model: BFL zWhy BFL instead of Attanasio, Meghir et Santiago (2005) ou Todd et Wolpin (2005)? ySimplicity since dynamic Ex ante models as AMS and TW are data intensive depending on panel data. yIs a behavioral model based on four key assumptions: xDo not model household behavioral, i.e., do not debate who makes child’s decision; xAdults are unafected by children’s choice; xSiblings interaction are ignored; xHousehold composition is exogeneous

11. 11 Ex ante model: BFL zThe model yChild’s occupational choice x(0) Not going to school; x(1) Going to school and paid work; x(2) Going to school and non-paid work

12. 12 Ex ante model: BFL zThe model yChild’s contribution to income in each state 0, 1 and 2

13. 13 Ex ante model: BFL zThe model yChild (household) i chooses the alternative that yields the highest utility

14. 14 Ex ante model: BFL zThe model yChild (household) i chooses the alternative that yields the highest simulated utility

15. 15 Ex ante estimator zAverage Intent to Treat effect (AIT) which provides an estimate of the average impact of the availability of the program to eligible households (in treatment communities) by simulating impact of the program on the sample of eligible age group of children; zAssumes good implementation of program zAttention: Ex ante model is static, i.e., no time or trend effects. zSo, it is best to compare AIT (ex ante) with AIT (ex post) obtained using 2DIF (which removes the trend effect from the estimated impact) whenever is possible.

16. 16 Results: PROGRESA

17. 17 Results: PROGRESA

18. 18 Results: PROGRESA

19. 19 Results: PROGRESA

20. 20 Conclusion zEx Ante model analysis indicates so far that they can be very useful as well as powerful in predicting program impacts. zBut work is still in progress. yUseful for simulating the design or re-design of a transfer program. yIncreasing demand from governments as Panama, Jamaica and Ecuador