Impact Evaluations and Social Innovation in Europe Bratislava, 15 December, 2011 Joost de Laat (PhD) Human Development Economics Europe and Central Asia The World Bank Comments:

PROGRESS 15 December 2011 Deadline

Inputs ActivitiesOutputs Impacts on Outcomes Finance State budget European Social Fund Human resources Min. of Labor Public employment offices Private/public training providers Project preparation activities (4 months) Identify 2000 long-term (at least 2 years) unemployed people interested in training program Identify 10 private / public training providers Design monitoring database Poject implementation activities (1 year). Offer training vouchers to 2000 unemployed Training institutes provide 3 months skills trainings to participating unemployed Enter data on training participation and job placement into database Improved labor market skills of current long-term unemployed Improved employment rates of current long-term unemployed Improved wage rates of current long-term unemployed Reduced poverty of current long-term unemployed Greater education enrolment among children of current long- term enrolment Improved health outcomes among children of current long- term unemployed. Project preparation outputs 2000 long term unemployed identified Contracts signed with 10 private/public training providers Monitoring database in place Project implementation 2000 vouchers allocated Est long term unemployed accept voucher and enlist in training Est long term unemployed complete training Est. 600 long term unemployed find jobs following training Database on participants

Outline What? Who? How? Ethics? Why? Impact Evaluations ?

What? Isolates causal impact on beneficiary outcomes Globally hundreds of randomized impact evaluations Canadian self-sufficiency welfare program Danish employment programs Turkish employment program India remedial education program Kenya school deworming program Mexican conditional cash transfer program (PROGRESA) United States youth development programs Different from: e.g. evaluation measuring whether social assistance is reaching the poorest households

Who? Often coalitions of: Governments International organizations Academics NGOs Private sector companies Examples: Poverty Action Lab (academic) Mathematica Policy Research (private) Development Innovations Ventures (USAID) International Initiative for Impact Eval. (3ie) WB Development IMpact Evaluation (DIME)

7 Ex 1: Development Innovation Venture (USAID)

Ex 2: International Initiative for Impact Evaluation (3ie)

9 Ex 3: MIT’s Poverty Action Lab (JPAL)

10 Ex 4: WB Development IMpact Evaluation (DIME)

11 Make publicly available training materials in partnership with other WB groups (e.g. Spanish Impact Evaluation Fund)

12 LocationDate Countries Attending ParticipantsProject Teams El Cairo, EgyptJanuary 13-17, Managua, NicaraguaMarch 3-7, Madrid, SpainJune 23-27, Manila, PhilippinesDecember 1-5, Lima, PeruJanuary 26-30, Amman, JordanMarch 8-12, Beijing, ChinaJuly 20-24, Sarajevo, BosniaSeptember 21-25, Cape Town, South AfricaDecember 7-11, Kathmandu, NepalFebruary 22-26, Total 861, Organize trainings on impact evaluations in partnership with others (e.g. Spanish Impact Evaluation Fund)

Impact Evaluation Clusters Conditional Cash Transfers Early Childhood Development Education Service Delivery HIV/AIDS Treatment and Prevention Local Development Malaria Control Pay-for-Performance in Health Rural Roads Rural Electrification Urban Upgrading ALMP and Youth Employment Help coordinate impact evaluations portfolio

How to carry one out? Basic Elements Comparison group that is identical at start of program Prospective: evaluation needs to be built into design from start Randomized evaluations generally most rigorous Example: randomize phase-in (who goes first?) Qualitative information – helps program design and understanding of the 'why'

Ethics? Implementation considerations Most programs cannot reach all: randomization provides each potential beneficiary fair chance of receiving program (early) Review by ethical review boards Broader considerations Important welfare implications of not spending resources effectively Is the program very beneficial? If we know the answer, there is no need for an IE

Why? EU2020 Targets (selected) 75% of the year-olds to be employed Reducing school drop-out rates below 10% At least 20 million fewer people in or at risk of poverty and social exclusion Policy Options Are Many Different ALMPs, trainings, pension rules, incentives for men taking on more home care etc. etc. For each policy options, also different intensities, ways of delivery…

Why? Selective Use of Impact evaluations Help provide answers to program effectiveness and design in EU2020 areas facing some of the greatest and most difficult social challenges But impact evaluations can also Build public support for proven programs Encourage program designers (govts, ngos, etc.) to focus more on program results Provide incentive to academia to focus energies on most pressing social issues like Roma inclusion!

Why? Help Encourage Social Innovation +

Thank you for your attention!