What can a CIE tell us about the origins of negative treatment effects of a training programme Miroslav Štefánik miroslav.stefanik(at)savba.sk INCLUSIVE GROWTH AND EMPLOYMENT IN EUROPE Bratislava

Motivation Data availability (Official registers of unemployed and Social insurance data) Critique of the training programme Counterfactual impact evaluation studies come in a wide stream of literature

Description of the training programme Training activities are implemented (also subcontracted) by regional offices of the Centre of Labour, Social Affairs and Family (COLSAF) The content of the trainings provided is widely defined (increasing employability) All registered unemployed are eligible to participate in the training, capacities are very limited Evaluation period Data allow us to follow the participants 24 months after the training – From 1/2007-4/2008 trainings were designed and organised (also subcontracted) by regional PES offices – From 5/2008 training providers are selected by a public procurement at the national level. Content of the trainings is decided based on the requests from regional offices (general skills). Bratislava remains out of this mechanism. – In 7/2010 new national projects are introduced with a rapid decline in numbers of participants and the accessibility of the trainings

Periods of implementation Source: Database on registered unemployed provided by COLSAF

Outcome indicators Working income – constructed from the assessed base of social insurance payments at the end of each month Employment – constructed using the information about the registration for social insurance payments (for each month)

Propensity score matching I- Participation in the training(0,1) X- vector of observed characteristics (all information available from the database): – Individual characteristics (gender, age, region, level and field of education,...) – Previous participation in other ALMM – Pre-treatment unemployment (date of entering, length and no. of previous unemployments,...) – Previous working experiences (days of previous working experience, economic sector and occupation,...) – Family background (kids, marital status,...) – Declared skills (PC skills, languages,...) Probit model to predict the propensity score variable (PSV)

PSM model applied: – 1:1 matching of the nearest neighbour – Replacement was allowed – Exact matching/Subgrouping based on regional offices – Two matching variables PSV The date of entering unemployment

Sensitivity analysis: PSM using caliper radius ( ) – Marginal improvement in balance – 46,6% of participants were excluded, leaving us with OLS estimation

Assumptions behind ex-post (control group selection) selection Unconfoundedness assumption After ensuring the balance on observable characteristics, non-participants outcomes have the same distribution that participants would have experienced if they had not participated. There are no unobservable characteristics influencing the outcome. Assumption of common support An area of common support exists=characteristics of participants and non-participants overlap. For each analysed participant, there is a non-participant which is sufficiently similar.

Distribution of the PSV before matching

PSV – Balance achievement N Log likelihood ,2 Prob > chi2 0,0000 Pseudo R2 0,5574 Sensitivity 28,24% Specificity 99,75% Positive predictive value 68,54% Negative predictive value 98,65% Correctly classified 98,42% Source: Database on registered unemployed and Social insurance database

Control groupParticipantsDatabaseBalance improvement N Mean mean (date of entry) ,84% mean(length of previous u) 511,36530,02312,5991,42% mean(age) 38, , , ,92% mean(psvar) 0, , , ,32% Proportion in % Male 45,2247,9754,1255,28% NP 9,7912,4836,4288,76% Single 37,3937,750,7797,63% Previous occupation ISCO 1 15,5717,5930,7084,59% ISCO 2 2,92,931,5897,78% ISCO 3 4,84,733,1895,48% ISCO 4 14,0713,997,6298,74% ISCO 5 8,047,64,784,83% ISCO 6 13,3613,711,6983,08% ISCO 7 0,580,620,9387,10% ISCO 8 15,4215,1213,2184,29% ISCO 9 15,1614,3717,2372,38% Foreign language 75,8576,0266,1998,27% Graduate 2,762,732,5484,21% Level of highest education achieved No elementary 0,090,080,5197,67% Elementary 18,5319,1524,1687,62% Lower socondary 0,43 1,07100,00% Vocational secondary 26,11 28,21100,00% Upper socondary vocational 39,2937,7230,0579,53% Upper secondary general 5,465,364,1291,94% First stage university 0,44 0,99100,00% Second stage university 21,0920,317,2474,18% Ph.D. 0,020,030,1490,91% Field of highest education achieved Field of education 1 19,2619,9426,2689,24% Field of education 2 0,340,530,64-72,73% Field of education 3 24,524,1721,9485,20% Field of education 4 17,317,1515,6889,80% Field of education 5 6,236,315,1892,92% Field of education 6 0,791,041,5146,81% Field of education 7 20,3620,5919,870,89% Field of education 8 9,148,587,5744,55%

Date of entering unemployment

Imputing the date of end of treatment for the control group Entering unemployment (Balanced) End of the treatment Number of days until the end of training Start of the reference period Entering unemployment (Balanced) Imputed end of the treatment Control group Participants

Results ATT on earnings: Comparison of methods Source: Database on registered unemployed and Social insurance database OLSPSM NN PSM Caliper MonthCoef.S.E.p.NCoef.S.E.pNCoef.S.E.pN 6-101,52,870, ,252,360, ,872,06660, ,643,190, ,442,960, ,802,49350, ,022,660, ,033,190, ,712,91410, ,773,630, ,23,540, ,07 6,970,

PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Employment) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

PSM estimations by period of implementation (Earnings) Source: Database on registered unemployed and Social insurance database

CBA scenarios Positive scenario ( ) Negative scenario ( ) Real scenario ( ) Additional employment Employment rate of participants Additional income Additional employment Employment rate of participants Additional income Additional employment Employment rate of participants Additional income 1. Year4,92%45,75%29,55-17,85%35,28%-130,16-7,75%40,23%-53,12 2. Year12,00%61,10%64,12-7,63%40,77%-97,52-3,14%54,60%-36,67 3+ years14,00%65,95%68,850,00%40,77%0,000,00%58,38%-35,00

CBA, 3 scenarios

Findings Evaluated training measure seems to have initial negative impact on participants chances to get employment and on their income The length of this initial (negative) impact varies between periods of implementation Positive impact of the measure is observed after 24 months (on average). In some periods of implementation positive impact is observable even earlier, in some periods there is none positive impact observable. Provided trainings seem to be less effective during and after the crisis. The way of implementation also plays a role in shaping the impact of the measure.

Thank you for your attention Miroslav Štefánik miroslav.stefanik(at)savba.sk