1. What Works? Evaluating the Impact of Active Labor Market Policies May 2010, Budapest, Hungary Joost de Laat (PhD), Economist, Human Development

2. Outline Why Evidence Based Decision Making? Active Labor Market Policies: Summary of Findings Where is the Evidence? The Challenge of Evaluating Program Impact Ex Ante and Ex Post Evaluation

3. 3 Why Evidence Based Decision Making? Limited resources to address needs Multiple policy options to address needs Rigorous evidence often lacking to prioritize policy options and program elements

4. 4 Active Labor Market Policies: Getting Unemployed into Jobs Improve matching of workers and jobs Assist in job search Improve quality of labor supply Business training, vocational training Provide direct labor incentives Job creation schemes such as public works

5. 5 Active Labor Market Policies Public Expenditure as % of GDP in OECD Countries, 2007 (OECD Stat) ACTIVE LABOR MARKET POLICIES 10: PES and administration 0.15 20: Training 0.14 30: Job rotation and job sharing 0.00 40: Employment incentives 0.10 50: Supported employment and rehabilitation 0.09 60: Direct job creation 0.05 70: Start-up incentives 0.01 TOTAL ACTIVE 0.56 PASSIVE LABOR MARKET POLICIES 80: Out-of-work income maintenance and support (incl. unemployment insurance) 0.64 90: Early retirement 0.11 TOTAL PASSIVE 0.75

6. 6 International Evidence on Effectiveness of ALMPs Active Labor Market Policy Evaluations: A Meta Analysis. By David Card, Jochen Kluve, and Andrea Weber (2009) Review of 97 studies between 1995-2007 The Effectiveness of European Active Labor Market Policy. By Jochen Kluve (2006) Review of 73 studies between 2002-2005

7. 7 Do ALMPs Help Unemployed Find Work? (Card et al. (2009), Kluve (2006)) Subsidized public sector employment Relatively Ineffective Job search assistance (often least expensive) Generally favorable, especially in short run Combined with sanctions (e.g. UK “New Deal”) promising Classroom and on-the-job training Not especially favorable in short-run More positive impacts after 2 years

8. 8 Do ALMPs Help Unemployed Find Work? (Card et al. (2009), Kluve (2006)) ALMPs targeted at youth Findings mixed

9. 9 The Impact Evaluation Challenge Impact is difference in outcome with and without program for those beneficiaries who participate in the program Problem: beneficiaries have only one existence; they participate in the program or they do not.

10. 10 Impact Evaluation Challenge: before – after comparison ok? beforeafter $1000 $2000 Skills Training Program Impact = $1000 extra income? Income for beneficiary increases from $1000 to $2000 after training

11. 11 Impact Evaluation Challenge: before – after often incorrect beforeafter $1000 $2000 NO Skills Training NO! Program Impact = $500 $1500 Income for the same person but without training would have increased from $1000 to $1500 because of improving economy

12. 12 Impact Evaluation Challenge Solution: a proper comparison group Comparison outcomes must be identical to treatment group outcomes, if the treatment group did not participate in the program.

13. 13 Impact Evaluation Approaches Ex ante: 1.Randomized evaluations 2.Double-difference (DD) methods Ex post: 3. Propensity score matching (PSM) 4. Regression discontinuity (RD) design 5. Instrumental variable (IV) methods

14. 14 Random assignment beforeafter $1000 $2000 Skills Training Program Impact = $500 $1500 Income comparison group is $1500 Income treatment group is $2000

15. 15 Randomized Assignment Ensures Proper Comparison Group Ensures treatment and comparison at start of program are the same (background and outcomes) Any differences that arise after program must be due to the program and not due to selection-bias “Gold” standard for evaluations; not always feasible

16. 16 Examples Randomized ALMP Evaluations Improve matching of workers and jobs Counseling the unemployed in France Improve quality of labor supply Providing vocationally focused training for disadvantaged youth in USA (Job Corps) Provide direct labor demand / supply incentives Canadian Self-Sufficiency Project

17. 17 Challenges to Randomized Designs Cost Ethical concerns: withholding a potentially beneficial program may be unethical Ethical concern must be balanced with: programs cannot reach all beneficiaries (and randomization may be fairest) knowing the program impact may have large potential benefits for society …

18. 18 Societal Benefits Rigorous findings lead to scale-up: Various US ALMP programs – funding by US Congress contingent on positive IE findings Opportunidades (PROGRESA) – Mexico Primary school deworming – Kenya Balsakhi remedial education – India

19. 19 Ongoing (Randomized) Impact Evaluations: From MIT Poverty Action Lab Website (2009)

20. 20 World Bank’s Development Impact Evaluation Initiative (DIME) 12 Impact Evaluation Clusters: Conditional Cash Transfers Early Childhood Development Education Service Delivery HIV/AIDS Treatment and Prevention Local Development Malaria Control Pay-for-Performance in Health Rural Roads Rural Electrification Urban Upgrading ALMP and Youth Employment

21. 21 Other Evaluation Approaches Ex ante: 1.Randomized evaluations 2.Double-difference (DD) methods Ex post: 3. Propensity score matching (PSM) 4. Regression discontinuity (RD) design 5. Instrumental variable (IV) methods

22. 22 Non-Randomized Impact Evaluations “ Quasi-experimental methods” Comparison group constructed by evaluator Challenge: evaluator can never be sure if behaviour of comparison group mimics that of treatment group without program: selection bias

23. 23 Example: Suppose Only Very Motivated Underemployed Seek Extra Skills Training Data on (very motivated) under-employed individuals who participated in skills training. Construct comparison group from (less motivated) under-employed who did not participate in skills training. DD method: evaluator compares increase in average incomes between two groups

24. 24 Double-Difference (DD) Method Treatment group Comparison group (non-randomization) Program impact (positive bias)

25. 25 Non-experimental design May provide unbiased impact answer Relies on assumptions regarding comparison Usually impossible to verify assumptions Bias always smaller if evaluator has detailed background variables (covariates)

26. 26 Assessing Validity of Non-Randomized Impact Evaluations Verify pre-program characteristics are same between treatment and comparison Test ‘impact’ of program on outcome variable that should not be affected by the program Note: will always hold in properly designed randomized evaluations

27. 27 Conclusion Everything else equal, experimental designs are preferred. Assess case-by-case. Most appropriate when: New program in pilot phase Not in pilot phase but receives large amounts of resources and its impact is questioned Non-experimental evaluations often cheaper; interpretation of results requires more scrutiny

28. 28 THANK YOU!

29. 29 Impact Evaluation Resources World Bank (2010) “Handbook of Impact Evaluations” by Khandker et al. www.worldbank.org/sief www.worldbank.org/dime www.worldbank.org/impactevaluation www.worldbank.org/eca/impactevaluation (last site coming soon) http://ec.europa.eu/regional_policy/sources/ docgener/evaluation/evaluation_en.htm http://ec.europa.eu/regional_policy/sources/ docgener/evaluation/evaluation_en.htm www.povertyactionlab.org http://evidencebasedprograms.org/