1. Impact evaluation of actions for jobseekers under the current OP ESF- Flemish Community : beyond classical parameters for success
Expert Hearing MS experiences on using control groups in ESF evaluations
25 October Brussel
Ludo Struyven, Liesbeth Van Parys, Greet Van Dooren & Bart Capéau
HIVA-K.U.Leuven

2. Expert Hearing Counterfactual Impact Evaluation
Overview
Quantitative Impact Evaluation of ALMPs
Economic literature
‘Soft skills’ as intermediary outcomes
Research questions
Comparison with previous evaluations
Theoretical causal chain model
Evaluation design
ESF actions + comparison group
Two-wave survey within strict period of time
Methodological aspects
Data collection arrangements
First results
Expert Hearing Counterfactual Impact Evaluation

3. Quantitative Impact Evaluation of ALMPs
State-of-the-art based on meta-analyses by (a.o.) Card et al. (2010)
It’s almost exclusively the programme type that matters for programme effectiveness
Direct employment programmes appear detrimental
‘Services and sanctions’ can be most effective
how to explain ?
Focus on how ALMPs improve one’s labour market chances
Expert Hearing Counterfactual Impact Evaluation

4. Expert Hearing Counterfactual Impact Evaluation
Economic literature
Theoretical perspectives
Lack of qualification > investment in human capital
Lack of motivation > disciplination strategy of those who face compulsory participation
Lack of information and networks > job search strategy
Current effect studies reflect very simplified perception of how labour markets work
Biased towards hard final outcomes such as time spent in (un)employment and labour market status x months after completion
Employability is left aside because ‘difficult to measure’
‘Bringing subjective factors back in’ : soft skills
Expert Hearing Counterfactual Impact Evaluation

5. Expert Hearing Counterfactual Impact Evaluation
‘Soft skills’
In this research three soft factors incorporated
criteria
proven to be crucial in previous studies
can be affected by each type of the ESF actions
relevant for participants with diverse distances from the labour market
Knowledge about the labour market
Job related self-knowledge
Job search related self-efficacy
Expert Hearing Counterfactual Impact Evaluation

6. Soft outcomes as intermediary outcomes ?
Existing studies on ‘soft outcomes’
Consider these factors as final outcomes, do not investigate the potential intermediary role of these outcomes
or they do include, but only consider the effect of the original level of these factors, not the effect of a change in the level thanks to participation in ALMP
Expert Hearing Counterfactual Impact Evaluation

7. Expert Hearing Counterfactual Impact Evaluation
Research questions
Main aim of our evaluation
To assess ‘soft outcomes’ in the short + medium term
To assess their impact as intermediary outcomes on final employment outcomes
Evaluation at micro level of ESF actions
Beyond classical parameters for success (employment outcome)
Recognition (and monitoring) of progress at individual level
Research questions
Does participation in each of the types of ESF actions has a positive effect on soft factors ?
Do the ESF actions foster participants’ chances to move into work only directly or also indirectly through their effect on soft outcomes ?
Expert Hearing Counterfactual Impact Evaluation

8. Comparison with previous evaluations of ESF programme
New policy choices
Compulsory comprehensive approach
An intervention is only completed when a person will move out of benefits
Control groups are not possible any more while in the past quasi-experimental designs with control groups were used
New ESF funding criteria
Funding of actions instead of interventions as a whole
Evaluation of actions while in the past the programme types (training, work experience, etc.) were evaluated (whether or not ESF financed)
Expert Hearing Counterfactual Impact Evaluation

9. Types of ESF actions + comparison group
Module 2: screening and orientation = reference group
Module 3: job application training
Module 4: vocational training
Module 5: person-oriented training
Module 6: support on the work floor
Module 7: follow-up interviews and coaching
ESF action = always sole or main action
Most of these actions are part of pathway offered to the unemployed
Expert Hearing Counterfactual Impact Evaluation

10. ‘quasi-experimental with nonequivalent groups’ design
Evaluation design
‘quasi-experimental with nonequivalent groups’ design
X2 O2
X3 O2
X4 O with:
Xi = treatment – indexes refer to numbers of modules
X5 O Oi= posttest observation
indicating that participants have not been randomly assigned
X6 O2
X7 O2

11. Theoretical causal chain model
Treatment
- ESF-action in module 2
- ESF-action in module 3
- ESF-action in module 4
- ESF-action in module 5
- ESF-action in module 6
- ESF-action in module 7
Context
- Personal characteristics
- Barriers to employment
- Characteristics of trajectory
Moderators
Intermediary outcomes: soft factors
- Knowledge of the labour market
- Job related self-knowledge
- Job search related self-efficacy
Final outcome:
Having moved into work x months after ESF-action
Social inclusion H1 + H2 H3
Expert Hearing Counterfactual Impact Evaluation

12. Two-wave survey within strict period of time
Stratified sample (n = ; 334/5 per action type)
1st survey = variable number of months after end action
2nd survey = fixed number of months after end action
Expert Hearing Counterfactual Impact Evaluation

13. Methodological aspects
Lack of control group
comparison group = ‘module 2’ actions
Lack of pretest-posttest design
self-reported retrospective assessment of changes in soft outcomes : ‘thank to participation …’
Selection bias due to unobserved characteristics
Participants’ propensity to be selected in each of the modules estimated by use of probit regression
Predicted values of latent variable based on 6 structural equations
Expert Hearing Counterfactual Impact Evaluation

14. Data collection arrangements
difficulties
solutions
no room to construct + test scales
Limited period of time to collect data in first survey
Difficulties in isolating ESF action
Classical parameter ‘moving into work’
Use of existing scales
Control for timelag between first survey + end of action
Control for other (not ESF) actions based on recorded data
Repeated recorded data and Job search behaviour +subjective factors (job quality, …) based on survey
Expert Hearing Counterfactual Impact Evaluation

15. First results (moving into work 6 m after completion of ESF action)
Direct effect
Indirect effect via job related self-knowledge
Indirect effect via knowledge labour market
Indirect effect via job search related self-efficacy
Total effect
Latent M2 - screening and orientation
0.3005
0.2708
1.5735
0.1471
Latent M3 - job application training
1.7129
0.2679
0.0153
Latent M4 - vocational training
0.0176
2.1716
0.3539
Latent M5 - person-oriented training
0.2967
2.0170
Latent M6 - support on the work floor
0.6737
0.1163
Latent M7 - follow-up interviews and coaching
2.1985
Expert Hearing Counterfactual Impact Evaluation

16. Thank you for your attention !
Main challenges
Refined measurement of employment outcome
Other relevant soft intermediary outcomes
Pretest-posttest design
Control for unobserved heterogenity
Thank you for your attention !
Prof. dr. Ludo Struyven – HIVA, KULeuven
Head of Unit, Labour Market Research
Parkstraat 47 (p.o. 5300) – B-3000 Leuven