Impact Evaluation for Evaluating Social Safety Nets at the World Bank Javier E. Baez Economist IEG World Bank AEA Conference San Antonio, TX November 12, 2010

Overview 1.Impact Evaluations – What are they? 2.Using Impact Evaluations to Inform Large Thematic Evaluations I.A Comprehensive Review of Existing Impact Evaluation Evidence in SSNs II.Conducting new IE in relevant areas with limited evidence 3.Conclusions

Impact Evaluations – What are they?

What is Impact Evaluation? ►Attribution: A method that seeks to identify the impact attributable to an intervention in contrast to just measuring changes in the variables of interest ►Counterfactual: Outcomes are compared with a counterfactual situation – what would have happened without the program ►Methods Techniques vary according to the program setting and available data, but can be broadly classified into two categories: (1) experimental, and (2) quasi-experimental

Using Impact Evaluations to Inform Large Thematic Evaluations

First exercise: a meta-review of IEs in Social Safety Nets … ►How? An exhaustive search, filtering and systematic organization of evidence from 137 completed IEs

to draw lessons about: (1) the short- term impacts of the programs, … ►Example: effects on school participation SSNs consistently increase the use of educational services and progression

(2) whether the impacts remain over time, … ►Example: effects on future incomes Some evidence suggests that SSNs increase incomes over time through various channels

(3) if the transfers of the programs change other behaviors, … ►Example: sexual, marriage and fertility decisions Some programs encouraged adolescent girls to: (1) adopt safer sexual behaviors and (2) delay early marriage and childbearing

and, (4) cases in which impacts may hold in different contexts ►Example for comparable conditional cash transfers programs across different settings: Impacts (direction and size) on some outcomes very stable across various contexts

Second exercise: Primary IEs of SSN programs ►Motivation: large knowledge gaps Even though SSNs are justified on the basis of their long-term objectives to reduce poverty, they are often evaluated only against short-term outcomes ►Goal: help fill in this gap and inform the broader evaluation IEG carried out ex-post IEs of two SSN programs in Pakistan and Colombia

The Familias en Acción CCT program in Colombia ►Main aspects of the program Cash transfers to 20 percent poorest households conditional upon certain behaviors (use of education and health services) Started in 2002 to help mitigate the effects of the economic crisis and protect investments in human capital Supported by the World Bank and the Inter-American Development Bank Expanded between ; covers approximately 2 million households

Does the program increase human capital? ►What did we know? Program increased school attendance and enrollment ►What did not we know? If increased school inputs actually translate into more human capital ►How did we get the answer? Assess the effects on high school completion by: –Using various sources of existing data –Exploiting the rules of eligibility to the program –Tracking registration in a national test given at the end of high school

Various sources of data were carefully merged to … Test scores of all students reaching 11 th grade (T and C) Administrative systems of the program (census of participants, only T) Poverty Census I (T and C)

track comparable participant and non- participant children over time...

and find that participants are more likely to finish high school ►Evidence is consistent and robust No jumps in other variables that should not be affected by the program (placebo test)

Conclusions

Do impact evaluation complement other evaluation approaches? ►Yes – Provides causal evidence for evaluation questions regarding “effectiveness” Can assess what works, why, for whom, and at what cost ►But be aware of its limitations Results may be program- and context-specific – especially in topics with little evidence accumulated Requires high quality data, could be expensive and take time to deliver findings to inform program and policy decisions ►Large thematic evaluations should be conceived to balance the strengths and limitations of each evaluation approach

Impact Evaluation for Evaluating Social Safety Nets at the World Bank Javier E. Baez Economist IEG World Bank AEA Conference San Antonio, TX November 12, 2010