1. Evaluation Partnership Meeting 12-13 March 2015
Using Counterfactual Impact Evaluations (CIE) for the Youth Employment Initiative (YEI)
Beatrice d’Hombres
Evaluation Partnership Meeting March 2015

2. What to take home? 1- CIE of YEI-funded interventions is feasable …only if you make NOW the necessary data arrangements 2 - Use administrative datasets for CIE whenever possible 3- Regression Discontinuity Design and Propensity Score Matching combined with a Difference in Differences are appropriate methods for CIE of YEI funded interventions 4 – Contact us if you need advice/help

3. Outline of the presentation
Why CIE again ?
Monitoring of the YEI : a step towards CIE
CIE and Counterfactual: data requirements
CIE in the context of the YEI: What do we want to evaluate?
CIE methods that could be applied to the YEI context
CIE Implementation: challenges
CRIE: what can CRIE do for you?

4. I - Why CIE again? …Mainly 3 reasons…

5. CIE is about quantifying the impact per se of a policy
Why CIE again?
1 - Common interest
Need to quantify the effect of the YEI: 6.4 billion Euro!
Why
Evidence-based policy needed to
Improve the quality and effectiveness of employment policies
Promote accountability
CIE is about quantifying the impact per se of a policy

6. Why CIE again? 2 – New requirements 2007-2013 2014-2020
Evaluation mainly on implementation issues
Two evaluations needed : 2015 and 2018
Regulations stress the importance of Impact
Evaluations

7. Why CIE again? 3 – Need to convince you NOW to make CIE a reality!
Plan and schedule evaluation
Make necessary data arrangements

8. II - Monitoring of the YEI : a step towards CIE
YEI in a snaphot
Monitoring : data requirements
13 November 2018

9. YEI in a snapshot
13 November 2018

10. Clear eligibility criteria & objective(s) : important for CIE
YEI in a snapshot
Clear eligibility criteria & objective(s) : important for CIE
Policy Framework
Youth Employment Package
Link with Youth Guarantee (YG)
Budget : € 6.4 billion
Eligibility criteria
Nuts2 Regions: Youth Unemployment rate
(YUR)>25% in 2012
NEETs, Individuals<25
Measures
Pathways/packages of measures with the long term (LT) objective of active labour market integration
SEE YEI COMMUNICATIONS
Eligibility criteria
Nuts2 Regions: Youth Unemployment rate (YUR)>25% in 2012
MS where YUR by more than 30% in 2012
Nuts 2 regions with YUR > 20% in 2012

11. II - Monitoring of the YEI : a step towards CIE
Monitoring of the YEI: data requirements
13 November 2018

12. Monitoring: data requirements
Indicators to be reported: financial, output, result indicators
Output indicators
Cover personal data: gender, employment status, age, educational level, household situation
Result indicators: related to participants only
Immediate result indicators: situation when the participant leaves the operation
All participants
Long term result indicators: 6 months after the participant left the operation
Based on representative samples
Micro data

13. Monitoring: data requirements
Long Term Indicators
Representative sample of treated participants
Store data in computerized form
Micro data
ID identifier
Intervention identifier
Treated individuals
Situation 6 months after participants left operation

14. CIE and Counterfactual: data requirements
Particularity of CIE with respect to other monitoring and evaluation tools?
CIE : data requirements

15. CIE and Counterfactual: data requirements
Particularity of CIE with respect
to other monitoring and evaluation tools?

16. CIE: quantifying the Impact
Impact (or net effect, treatment effect) of an intervention – “the contribution of a certain policy to the change in the result indicator" (Gaffey (2012), p. 6)
CIE: particular type of evaluation, seek to answer cause-and-effect questions
CIE: one tool among others
Must be associated with a theory-based impact evaluation
Must be complemented by monitoring/process evaluation
Limits of CIE
Examines "the effect of the cause“ but does not explain "the causes of an effect“
Can be only applied to some specific policies/interventions

17. Ingredients for a CIE
Well defined intervention targeted at a well defined population Some units i are exposed to the policy intervention treated unit others are not untreated/control units Di=1 if treated Di=0 if untreated Expected effect: change of the outcome Yi for the units i exposed to the intervention Yi=Y1i if treated Yi=Y0i =if untreated

18. Ingredients for a CIE
The treatment effect for individual i is defined by Y1i- Y0i Problem: only one potential outcome is observed which implies that we cannot measure Y1i- Y0i Consequence: we must build a counterfactual to be able to estimate the impact of the treatment Counterfactual: if we observe Y1i in the treatment group, we need to estimate Y0i , the “value of the outcome variable for individual i in the treatment group had it been assigned to the control group instead'

19. CIE in practice
Individual causal effects are impossible to estimate, hence
we rely on average treatment effects
Treated units: average outcome Ave(Y1i)
Control units used for the counterfactual:
average outcome Ave(Y0i)
Impact of the intervention: Ave(Y1i) -Ave(Y0i)
Data: it is necessary to have information on treated and untreated units

20. CIE: Data Requirements
Data with information on treated and not treated individuals
Just looking at the change in the employment/education status of participants is not enough to measure the effect of the intervention
Need to find a control group

21. CIE: Data Requirements
CIE objective: identify a good comparison group to calculate a summary measure of the impact of the intervention
Without a control group, it is impossible to quantify the impact of the intervention
Bad control group: biased estimate of the impact of the intervention because of the selection bias
Constructing the control group can be challenging especially in terms of data access because of legal and other barriers

22. CIE – How to collect information for the control group?
Administrative datasets when possible
Population, tax, unemployment records
Link the datasets together using ID identifier (social security number for instance)
Link with the data from YEI funded interventions providers (PES, educational structures, etc.)
Follow-up surveys to collect data on long term outcomes (if different from those already included in administrative datasets)

23. CIE in the context of the YEI
What do we want to evaluate ?
Methods to identify a proper control group

24. CIE in the context of the YEI
What do we want to evaluate ?

25. What do we want to evaluate?
Objective
Sustainable integration
in the labor (LM) of the NEETs
In practice
Labour market outcomes after a certain period of time (6 months, 12, 24 months)
Employment status
Quality of the job (remuneration level, part time job, open-ended contracts, relevance of the job, etc)
Education and training related outcomes after a certain period of time (6 months, 12, 24 months
Evidence on SH exists. Scant evidence on the LT effects

26. What do we want to evaluate?
1 Are YEI funded interventions a success story in roughly all its components for the integration of NEETS on the LM?
2 Effect of a specific YEI interventions (effect of start up support for young entrepreneurs for instance)
3 Effect of alternative YEI interventions (reduction of non wage labour costs versus wage & recruitement subsidies)

27. CIE in the context of the YEI
Methods to identify a proper control group

28. CIE – Methods to identify a good counterfactual
Experimental approach
Quasi-experimental methods
Regression Discontinuity Design (RDD)
Propensity Score Matching (PSM)
Difference in Differences (DiD)
Instrumental Variables (IV)
Choice of the method(s) depends on:
Eligibility criteria to the intervention
Data availability

29. Regression Discontinuity Design - RDD
CIE of the YEI
Regression Discontinuity Design - RDD

30. RDD: intuition
Eligibility for a program determined by a rule treatment assignment based on the value of a continuous variable Si called the forcing variable
Treatment=1 if Si<c
Treatment =0 if Si>=c
Participation to the program is not voluntary
In the YEI context
Si : Age
c=25
Yi: LM outcome

31. RDD: intuition Without the intervention, linear case
In the YEI context
Si : Age
c=25
Yi: LM outcome

32. RDD: intuition After the intervention, linear case In the YEI context
Si : Age
c=25
Yi: LM outcome
Possiblity to extend the target group, NEET under the age of 30
There are not other policies (in addition to the YEI intervention) using the same eligibility criteria and which will affect the outcome

33. RDD: intuition After the intervention, linear case In the YEI context
Si : Age
c=25
Yi: LM outcome
Possiblity to extend the target group, NEET under the age of 30
Need of a large sample size

34. RDD: intuition
Eligibility rule: On either sides of c, individuals have very similar characteristics, but some are treated and others are not
Counterfactual: units above the cut-off who did not participate, i.e marginal non beneficiaries
Effect of the intervention: difference in the average performance between marginal beneficiaries and marginal non beneficiaries
In the YEI context
Marginal beneficiaries
NEETs below 25
Marginal non beneficiaries NEETS 25 or more

35. RDD: Example of an application (1)

36. RDD: Example of an application (1)
Implementation of a Youth Guarantee (YG) in 1994
Obligation to offer unemployed aged a full time activity after 100 days of unemployment
Duration of the program capped at 12 months
Major programmes: labour market training and practice program
YG also includes job search assistance/counselling
Impact of the YG?
Possiblity to extend the target group, NEET under the age of 30

37. RDD: Example of an application (1)
Administrative data
Handel database: data on all individuals registered at the PES including time of registration, program placement reason for leaving registers, some background characteristics
Statistics Sweden: information about earnings, welfare receipts, etc

38. RDD: Example of an application (1)
Short term impact
Outflow to work from unemployment registers

39. RDD: Exemple of an application (1)

40. RDD: Example of an application (2)
Presented at COMPIE

41. RDD: Example of an application (2) YG in Finland
Before 2005
From 2005 on
PES offered similar services for the youths and the adults
No draft job-search plan
Signed job-search plan within 5 months
NO Guarantee
Specific service for youths (below 25)
Draft job-search plan within 1 month
Signed job-search plan within 3months
Activation measures
Activation measures: subsidized employment, labour market training , work practise etc.
Possiblity to extend the target group, NEET under the age of 30

42. RDD: Example of an application (2) YG in Finland
Register-based data on individuals over the period of
e.g. Population Register Centre, Local Register Offices, Tax Administration, Social Insurance Institution, National Institute for Health and Welfare, Centre for Pensions, Labour administration, National Board of Education
20 per cent random sample of birth cohorts
Each year a 20 per cent random sample of new entrants in the population register is added to the sample
Identification based on the age limit of 25 years
Assume: no self-selection between the treated and the controls
Possiblity to extend the target group, NEET under the age of 30

43. RDD: Example of an application (2)
Heterogeneity of the results according to the level of education – the unemployed youth (aged 23-27)

44. RDD – Wrap up
Possible if assignment to the program only determined by a rule
If probability to be treated <1 if NEETs below 25 (capacity issues), then sharp RDD measures the intention to treat effect
No sharp RDD if participation also depends on individual characteristics (motivation, priority based on individual characteristics); in this case, other CIE methods need to be used
No other policies using the same eligibility criteria and which will affect the outcome
Possiblity to extend the target group, NEET under the age of 30

45. CIE – How to identify a counterfactual?
Experimental approach
Quasi-experimental methods
Regression Discontinuity Design (RDD)
Propensity Score Matching (PSM)
Difference in Differences (DiD)
Instrumental Variables (IV)
Choice of the method(s) depends on
Eligibility criteria to the intervention
Data availability

46. Propensity Score Matching - PSM
CIE of the YEI
Propensity Score Matching - PSM

47. PSM: under which conditions?
Applied when participation to a program is in part determined by the decision of the units (self-selection)
Example: YG offer: training versus apprenticeship, choice in part decided by the participants
Match participants to the program with non participants sharing similar observable characteristics
Previous labour market experience,
Age, education
Gender, family composition, etc.
Crucial difference with a sharp RDD: match based on observable characteristics and not an exogeneous assignment rule

48. PSM: Intuition
Ideally match each treated unit with a non treated one with exactly the same characteristics relevant for the selection process
Problem: As the number of characteristics determining selection increases it is more and more difficult to find
comparable individuals): curse of dimensionality
Propensity Score Matching: Matching on a single index (propensity score, PS) reflecting the probability of participation.

49. PSM Intuition
Propensity score
Probability of participating in the intervention, conditional on the characteristics

50. PSM Intuition
Counterfactual: non participants with similar propensity score characteristics as the participants
Matching based on the propensity score

51. PSM Intuition
Different ways of matching treated with non treated units
Impact : average of the difference in the post treatment period between each treated and matched units

52. Combine PSM and Difference in Differences (DiD)
PSM valid if and only if no systematic differences in unobservable characteristics that influence the outcome and the participation to the program (motivation, ability, etc)
Possible to combine PSM with the DiD method to remove any time invariant characteristic
DiD: graphically

53. PSM and DiD Impact = B-A Possible to combine PSM with DiD PSM
Impact : average of the difference in the post treatment period between each treated and matched units
PSM + DiD
A- Compute average of the difference in the pre-treatment period between each treated and matched units
B- Compute average of the difference in the post treatment period between each treated and matched units
Impact = B-A

54. When does PSM apply in the YEI context?
Impact of the YEI funded interventions (set of measures)
Treatment based on elegiblity criteria and case worker/participant decisions
Impact of the alternative activation measures
Conditional on being eligible to the YEI interventions, treatment (specific activation measure) based on case worker/participant decisions

55. PSM: Example of an application

56. PSM: Example of an application (1)
Implementation of a Youth Guarantee (YG) in 1994
Obligation to offer unemployed aged a full time activity after 100 days of unemployment
Duration of the program capped at 12 months
Major programmes: labour market training and practice program
YG also includes job search assistance/counselling
Impact of labour market training versus practice program?

57. PSM: Example of an application (1)
Administrative data
Handel database: data on all individuals registered at the PES including time of registration, program placement reason for leaving registers, some background characteristics
Statistics Sweden: information about earnings, welfare receipts, etc

58. PSM: Example of an application (1)
Variables used for the PSM
Education, family situation, labour market history, demographic characteristics
Assumption
Conditional on the variables above, the only difference between treated and non treated participants is the treatment status

59. PSM: Example of an application (1)

60. PSM– Wrap up
Data Hungry method: need of datasets with many variables on the characteristics of the individuals
Assumption that no unobserved confounding factors affect the outcome enhanced by incorporation of a rich range of variables into the estimation of the propensity scores
Possibility to combine a PSM with a DiD to remove time invariant unobserved characteristics such as motivation
When?
Rich dataset
Participation to YEI funded interventions also depends on participants/caseworker decisions

61. CIE Implementation - Challenges

62. CIE Implementation - Challenges
Starting point: policy questions and hypotheses to be tested.
Evaluation design operationalization:
Evaluation design: define comparison group; rules of programme operation  (but not ):CIE method.
Evaluation team: partnerships (policy makers + private/public agencies).
Evaluation timing and budget definition  data collection.
Outsourcing vs. in-house CIEs: can outsource any of
sample definition, data collection and CIE implementation
but crucial to develop in-house CIE knowledge.

63. CRIE: what can we do for you?
Training Collaboration

64. CRIE: what can we do for you?
1 - Regional Workshops   Paper and Pencil Training on CIE methods Computer-based training Presentation and discussion of CIE performed by MA Practical sessions to design CIE The RWC must involve several Member States Share of good practices of CIE performed by peers Need that EMPL+JRC & MS work as a team to make CIE happen
Need for building capacity: CIE Trainings in PL, ES, HU, CZ, BG, EE, SK, LV, LT

65. CRIE: what can we do for you? Possible data analysis collaboration
2- Collaboration with the quantitative analysis
Example: Latvia Finance Ministry
Possible data analysis collaboration

66. What to take home? 1- CIE of YEI-funded interventions is feasable …only if you make NOW the necessary data arrangments 2 - Use administrative datasets for CIE whenever possible 3- Regression Discontinuity Design and Propensity Score Matching combined with a Difference in Differences are appropriate methods for CIE of YEI funded interventions. 4 – Contact us if you need advice/help

67. For more information and contacts: http://crie.jrc.ec.europa.eu