1. What can a CIE tell us about the origins of negative treatment effects of a training programme Miroslav Štefánik miroslav.stefanik(at)savba.sk INCLUSIVE GROWTH AND EMPLOYMENT IN EUROPE 3.11. 2015 Bratislava

2. Motivation Data availability (Official registers of unemployed and Social insurance data) Critique of the training programme Counterfactual impact evaluation studies come in a wide stream of literature

3. Description of the training programme Training activities are implemented (also subcontracted) by regional offices of the Centre of Labour, Social Affairs and Family (COLSAF) The content of the trainings provided is widely defined (increasing employability) All registered unemployed are eligible to participate in the training, capacities are very limited Evaluation period 2007-2013 Data allow us to follow the participants 24 months after the training – From 1/2007-4/2008 trainings were designed and organised (also subcontracted) by regional PES offices – From 5/2008 training providers are selected by a public procurement at the national level. Content of the trainings is decided based on the requests from regional offices (general skills). Bratislava remains out of this mechanism. – In 7/2010 new national projects are introduced with a rapid decline in numbers of participants and the accessibility of the trainings

4. Periods of implementation Source: Database on registered unemployed provided by COLSAF

5. Outcome indicators Working income – constructed from the assessed base of social insurance payments at the end of each month Employment – constructed using the information about the registration for social insurance payments (for each month)

6. Propensity score matching I- Participation in the training(0,1) X- vector of observed characteristics (all information available from the database): – Individual characteristics (gender, age, region, level and field of education,...) – Previous participation in other ALMM – Pre-treatment unemployment (date of entering, length and no. of previous unemployments,...) – Previous working experiences (days of previous working experience, economic sector and occupation,...) – Family background (kids, marital status,...) – Declared skills (PC skills, languages,...) Probit model to predict the propensity score variable (PSV)

7. PSM model applied: – 1:1 matching of the nearest neighbour – Replacement was allowed – Exact matching/Subgrouping based on regional offices – Two matching variables PSV The date of entering unemployment

8. Sensitivity analysis: PSM using caliper radius (0.00075) – Marginal improvement in balance – 46,6% of participants were excluded, leaving us with 21 288 OLS estimation

9. Assumptions behind ex-post (control group selection) selection Unconfoundedness assumption After ensuring the balance on observable characteristics, non-participants outcomes have the same distribution that participants would have experienced if they had not participated. There are no unobservable characteristics influencing the outcome. Assumption of common support An area of common support exists=characteristics of participants and non-participants overlap. For each analysed participant, there is a non-participant which is sufficiently similar.

10. Distribution of the PSV before matching

11. PSV – Balance achievement N 1 758 123 Log likelihood 181 862,2 Prob > chi2 0,0000 Pseudo R2 0,5574 Sensitivity 28,24% Specificity 99,75% Positive predictive value 68,54% Negative predictive value 98,65% Correctly classified 98,42% Source: Database on registered unemployed and Social insurance database

12. Control groupParticipantsDatabaseBalance improvement N 32.651 2.354.850 Mean mean (date of entry) 25.12.0826.12.082.9.1099,84% mean(length of previous u) 511,36530,02312,5991,42% mean(age) 38,4967438,3255334,9579594,92% mean(psvar) 0,41489810,41730190,062217899,32% Proportion in % Male 45,2247,9754,1255,28% NP 9,7912,4836,4288,76% Single 37,3937,750,7797,63% Previous occupation ISCO 1 15,5717,5930,7084,59% ISCO 2 2,92,931,5897,78% ISCO 3 4,84,733,1895,48% ISCO 4 14,0713,997,6298,74% ISCO 5 8,047,64,784,83% ISCO 6 13,3613,711,6983,08% ISCO 7 0,580,620,9387,10% ISCO 8 15,4215,1213,2184,29% ISCO 9 15,1614,3717,2372,38% Foreign language 75,8576,0266,1998,27% Graduate 2,762,732,5484,21% Level of highest education achieved No elementary 0,090,080,5197,67% Elementary 18,5319,1524,1687,62% Lower socondary 0,43 1,07100,00% Vocational secondary 26,11 28,21100,00% Upper socondary vocational 39,2937,7230,0579,53% Upper secondary general 5,465,364,1291,94% First stage university 0,44 0,99100,00% Second stage university 21,0920,317,2474,18% Ph.D. 0,020,030,1490,91% Field of highest education achieved Field of education 1 19,2619,9426,2689,24% Field of education 2 0,340,530,64-72,73% Field of education 3 24,524,1721,9485,20% Field of education 4 17,317,1515,6889,80% Field of education 5 6,236,315,1892,92% Field of education 6 0,791,041,5146,81% Field of education 7 20,3620,5919,870,89% Field of education 8 9,148,587,5744,55%

13. Date of entering unemployment

14. Imputing the date of end of treatment for the control group Entering unemployment (Balanced) End of the treatment Number of days until the end of training Start of the reference period Entering unemployment (Balanced) Imputed end of the treatment Control group Participants

15. Results ATT on earnings: Comparison of methods Source: Database on registered unemployed and Social insurance database OLSPSM NN PSM Caliper MonthCoef.S.E.p.NCoef.S.E.pNCoef.S.E.pN 6-101,52,870,0001757898-20,252,360,00060168-49,872,06660,00041380 12-82,643,190,0001757805-16,442,960,00059889-38,802,49350,00041380 18-29,022,660,0001757296-3,033,190,343358907-25,712,91410,00041380 2430,773,630,000175672913,23,540,000257802-7,07 6,970,311 41380

16. PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

17. PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

18. PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

19. PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

20. PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

21. PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

22. PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

23. PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

24. PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

25. PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

26. PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

27. PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

28. PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

29. PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

30. CBA scenarios Positive scenario (200701- 200804) Negative scenario (201201- 201212) Real scenario (200701- 201312) Additional employment Employment rate of participants Additional income Additional employment Employment rate of participants Additional income Additional employment Employment rate of participants Additional income 1. Year4,92%45,75%29,55-17,85%35,28%-130,16-7,75%40,23%-53,12 2. Year12,00%61,10%64,12-7,63%40,77%-97,52-3,14%54,60%-36,67 3+ years14,00%65,95%68,850,00%40,77%0,000,00%58,38%-35,00

31. CBA, 3 scenarios

32. Findings Evaluated training measure seems to have initial negative impact on participants chances to get employment and on their income The length of this initial (negative) impact varies between periods of implementation Positive impact of the measure is observed after 24 months (on average). In some periods of implementation positive impact is observable even earlier, in some periods there is none positive impact observable. Provided trainings seem to be less effective during and after the crisis. The way of implementation also plays a role in shaping the impact of the measure.

33. Thank you for your attention Miroslav Štefánik miroslav.stefanik(at)savba.sk