Impact evaluation of actions for jobseekers under the current OP 2007-2013 ESF- Flemish Community : beyond classical parameters for success Expert Hearing MS experiences on using control groups in ESF evaluations 25 October 2011 - Brussel Ludo Struyven, Liesbeth Van Parys, Greet Van Dooren & Bart Capéau HIVA-K.U.Leuven

Expert Hearing Counterfactual Impact Evaluation Overview Quantitative Impact Evaluation of ALMPs Economic literature ‘Soft skills’ as intermediary outcomes Research questions Comparison with previous evaluations Theoretical causal chain model Evaluation design ESF actions + comparison group Two-wave survey within strict period of time Methodological aspects Data collection arrangements First results 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

Quantitative Impact Evaluation of ALMPs State-of-the-art based on meta-analyses by (a.o.) Card et al. (2010) It’s almost exclusively the programme type that matters for programme effectiveness Direct employment programmes appear detrimental ‘Services and sanctions’ can be most effective how to explain ? Focus on how ALMPs improve one’s labour market chances 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

Expert Hearing Counterfactual Impact Evaluation Economic literature Theoretical perspectives Lack of qualification > investment in human capital Lack of motivation > disciplination strategy of those who face compulsory participation Lack of information and networks > job search strategy Current effect studies reflect very simplified perception of how labour markets work Biased towards hard final outcomes such as time spent in (un)employment and labour market status x months after completion Employability is left aside because ‘difficult to measure’ ‘Bringing subjective factors back in’ : soft skills 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

Expert Hearing Counterfactual Impact Evaluation ‘Soft skills’ In this research three soft factors incorporated criteria proven to be crucial in previous studies can be affected by each type of the ESF actions relevant for participants with diverse distances from the labour market Knowledge about the labour market Job related self-knowledge Job search related self-efficacy 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

Soft outcomes as intermediary outcomes ? Existing studies on ‘soft outcomes’ Consider these factors as final outcomes, do not investigate the potential intermediary role of these outcomes or they do include, but only consider the effect of the original level of these factors, not the effect of a change in the level thanks to participation in ALMP 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

Expert Hearing Counterfactual Impact Evaluation Research questions Main aim of our evaluation To assess ‘soft outcomes’ in the short + medium term To assess their impact as intermediary outcomes on final employment outcomes Evaluation at micro level of ESF actions Beyond classical parameters for success (employment outcome) Recognition (and monitoring) of progress at individual level Research questions Does participation in each of the types of ESF actions has a positive effect on soft factors ? Do the ESF actions foster participants’ chances to move into work only directly or also indirectly through their effect on soft outcomes ? 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

Comparison with previous evaluations of ESF programme New policy choices Compulsory comprehensive approach An intervention is only completed when a person will move out of benefits Control groups are not possible any more while in the past quasi-experimental designs with control groups were used New ESF funding criteria Funding of actions instead of interventions as a whole Evaluation of actions while in the past the programme types (training, work experience, etc.) were evaluated (whether or not ESF financed) 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

Types of ESF actions + comparison group Module 2: screening and orientation = reference group Module 3: job application training Module 4: vocational training Module 5: person-oriented training Module 6: support on the work floor Module 7: follow-up interviews and coaching ESF action = always sole or main action Most of these actions are part of pathway offered to the unemployed 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

‘quasi-experimental with nonequivalent groups’ design Evaluation design ‘quasi-experimental with nonequivalent groups’ design X2 O2 -------------- X3 O2 X4 O2 with: -------------- Xi = treatment – indexes refer to numbers of modules X5 O2 Oi= posttest observation -------------- --- indicating that participants have not been randomly assigned X6 O2 X7 O2

Theoretical causal chain model Treatment - ESF-action in module 2 - ESF-action in module 3 - ESF-action in module 4 - ESF-action in module 5 - ESF-action in module 6 - ESF-action in module 7 Context - Personal characteristics - Barriers to employment - Characteristics of trajectory Moderators Intermediary outcomes: soft factors - Knowledge of the labour market - Job related self-knowledge - Job search related self-efficacy Final outcome: Having moved into work x months after ESF-action Social inclusion H1 + H2 H3 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

Two-wave survey within strict period of time Stratified sample (n = 2.005 ; 334/5 per action type) 1st survey = variable number of months after end action 2nd survey = fixed number of months after end action 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

Methodological aspects Lack of control group comparison group = ‘module 2’ actions Lack of pretest-posttest design self-reported retrospective assessment of changes in soft outcomes : ‘thank to participation …’ Selection bias due to unobserved characteristics Participants’ propensity to be selected in each of the modules estimated by use of probit regression Predicted values of latent variable based on 6 structural equations 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

Data collection arrangements difficulties solutions no room to construct + test scales Limited period of time to collect data in first survey Difficulties in isolating ESF action Classical parameter ‘moving into work’ Use of existing scales Control for timelag between first survey + end of action Control for other (not ESF) actions based on recorded data Repeated recorded data and Job search behaviour +subjective factors (job quality, …) based on survey 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

First results (moving into work 6 m after completion of ESF action) Direct effect Indirect effect via job related self-knowledge Indirect effect via knowledge labour market Indirect effect via job search related self-efficacy Total effect Latent M2 - screening and orientation 0.3005 -1.9977 0.2708 1.5735 0.1471 Latent M3 - job application training 1.7129 0.2679 0.0153 -2.0566 -0.0604 Latent M4 - vocational training -0.4340 -1.4013 0.0176 2.1716 0.3539 Latent M5 - person-oriented training 0.2967 2.0170 -0.2319 -2.5668 -0.4850 Latent M6 - support on the work floor 0.6737 -0.1983 -0.0526 -0.3065 0.1163 Latent M7 - follow-up interviews and coaching -0.0341 2.1985 -0.1086 -2.1532 -0.0974 25-10-2011 Expert Hearing Counterfactual Impact Evaluation

Thank you for your attention ! Main challenges Refined measurement of employment outcome Other relevant soft intermediary outcomes Pretest-posttest design Control for unobserved heterogenity Thank you for your attention ! Prof. dr. Ludo Struyven – HIVA, KULeuven Head of Unit, Labour Market Research Parkstraat 47 (p.o. 5300) – B-3000 Leuven Ludo.struyven@hiva.kuleuven.be