1. Centre for Research on Impact Evaluation
CIE in Structural Funds evaluation: challenges and opportunities The CRIE experience and perspective
Centre for Research on Impact Evaluation
(CRIE, DG-JRC)
Paolo Paruolo
Joint ESF Evaluation Partnership and ERDF Evaluation Network, Athens 15 May 2014

2. CIE challenges & opportunities CRIE experience & perspective
Outline
CIE challenges & opportunities
CRIE experience & perspective
 important reference: DG EMPL 2013 Guidance Document
“Design and commissioning of counterfactual impact evaluation”

3. CIE challenges: Feasibility issues
Is the intervention feasible for CIE?
Operational and methodological aspects:
Intervention features:
Innovative and policy relevant?
Distinctive and homogeneous?
Theory of change?
CIE characteristics:
Cost-effective? Value-added?
Target group/population?
Control/comparison group/population?
Eligibility vs. participation (mandatory? randomization?)

4. CIE challenges: Data collection
Data availability dictates if/how
a CIE can be implemented:
Sample selection and Data collection:
Plan in advance: treat. status, outcome, control variables.
Control group information crucial
 possibly gather it before program starts.
Microdata: single eligible units level vs. aggregated data.
Data sources: Monitoring data is insufficient
Administrative data: contact statistical/governmental offices
 merge alternative data sources.
Survey data: pre/post intervention surveys

5. CIE challenges: Implemention
Starting point: policy questions and hypotheses to be tested.
Evaluation design operationalization:
Evaluation design: define comparison group; rules of programme operation  (but not ):CIE method.
Ethics requirements.
Evaluation team: partnerships (policy makers + privates/public agencies).
Evaluation timing and budget definition  data collection.
Outsourcing vs. in-house CIEs: can outsource any of
sample definition, data collection and CIE implementation
but crucial to develop in-house CIE knowledge.
CRIE contributes to this build-up of capacity

6. CRIE’s experience CRIE: Centre for Research on Impact Evaluation
Joint DG EMPL-DG JRC initiative
Established in June 2013
Support to MS and DG EMPL to set up the necessary arrangements for carrying out Counterfactual Impact Evaluations (CIE) of ESF funded interventions
Scientists with a background in Economics and Statistics

7. CRIE Activities 1. Support to DG EMPL 2. Support to MS
3. Research activities

8. Support to DG EMPL Knowledge gaps from the academic literature
Intervention type
1. Training
2. Employment Incentives
3. Labor Market Services (LMS)
MS implementing CIEs*
MS-level Knowledge gaps**
MS implementing CIEs
MS-level Knowledge gaps
Target group
Unemployed
AT(1), DE(21), DK(1), IE(1), LV(1), PL(3), RO(1), SE(4), SI(1) 19
DE(10), IT(2), SE(1), FI(1) 24
DK(3), RO(1), IE(1), NL(1), SE(1), DE(1), AT(1) 20
Young unemployed
FR(1), SE(1), UK(2)
DE(2), FR(1), IT(1), PL(1), UK(1), SE(1) 21
FR(1), PT(1), SE(1), UK(2) 23
Disadvantaged young unemployed
DE(1), FR(1) 25
DE(1) 26 - 27
Old unemployed
PT(1)
Long-term unemployed
DE(6), BG(1), SE(2), PL(1)
FR(1), SE(1), UK(1)
Low-skilled unemployed
DK(1)
Employed
FR(1)
FI(1), ES(1), PL(1), DE(1)
Inactive
IT(1)
Disabled
* country for which there is CIEs evidence (number of papers)
** number of country-level knowledge gaps (i.e. # countries not implementing CIEs)

9. Knowledge gaps from the academic literature
1. Training
23 papers for DE interventions; 5 for SE; newer MS severely under-represented.
34 papers evaluating interventions aimed at unemployed.
Disadvantaged young unemployed, long-term unemployed, low-skilled unemployed, employed, disabled categories have 1 to 3 entries → lack of evidence.
2. Employment incentives
Majority of CIE evaluations in DE and targeting unemployed, long term unemployed and young unemployed.
Lack of evidence for Southern and Eastern Europe.
Outcome. P(e), Job stability, earnings 3 months after completing : short term effect
Eligibility: registered in PEO, below 30 and specific educational levels
Insturment : age (fuzzy rdd- though not clear if fuzzy)
Other iv: change in amount of grant- duration of internship (likely effect on outcome)

10. Knowledge gaps from the academic literature
3. Labour Market Services
Less CIEs as compared to Training and Employment incentives.
Unemployed again the most frequent target group; young unemployed also high.
No CIE evidence for inactive and vulnerable groups.
No concentration of interventions in some particular EU countries. Highest CIE numbers for DK and UK.
With the exception of RO, no evidence for east European MS.
Outcome. P(e), Job stability, earnings 3 months after completing : short term effect
Eligibility: registered in PEO, below 30 and specific educational levels
Insturment : age (fuzzy rdd- though not clear if fuzzy)
Other iv: change in amount of grant- duration of internship (likely effect on outcome)

11. CRIE’s perspective More CIEs are needed
They are feasible, especially within the new programming period

12. Support to MS: CIE training workshops
Mainly two types:
1 day or 2-3 day modules
Member State
Date
Training type
Poland
Nov. 2013
2 days
Spain
Dec. 2013
1/2 day intro.
Hungary
Jan. 2014
1 day intro.
Czech Republic
Bulgaria
Apr. 2014
3 days
Estonia
Slovak Republic
21-23 May
Latvia
3-5 June
Lithuania
To be decided
Belgium
Past and future trainings

13. CIE training workshops
Support to MS:
CIE training workshops
Example of a 2-day training workshop
Schedule:
Day1
Introduction – Randomized control trials, with examples
Propensity score matching method, with examples
Difference in difference method, with examples
Group exercise
Day2
Regression discontinuity design, with examples
Instrumental variables and combination of methods, with examples
Practical considerations in preparing a CIE

14. Trainings: satisfaction of participants:

15. Trainings: satisfaction of participants

16. Participants’ suggestions:
Longer course
(requested also by 3-day training participants)
More real-world examples
and exercises and more
details on practical/methodological issues.
Computer software course on CIE methods

17. CRIE’s perspective on Training Workshops
Need for a continued support to MS
over the programming period.
Future evolution:
Training workshops
Computer-based Training workshops
Workshops with training tutorials
Workshops with discussions
(at various
geographical levels).

18. Support to MS: direct support Possible data analysis collaboration
Tailor-made advice on methodological and data issues that will arise while implementing the CIE
Example: Pilot kick-off meetings, Social Experimentation kick-off meetings, …
Collaboration with the quantitative analysis
Example: Latvia Finance Ministry delegation (March 2014):
2-day presentation in Ispra
Possible data analysis collaboration

19. CRIE’s perspective on collaborations with MS
Evaluation of past intervention is difficult because retrieving the right data is not so obvious: need to plan ahead
Merging data sets is the only available option for the ; this need not be the case in the programming period, when you have more options

20. promotion of a CIE culture
Support to MS:
promotion of a CIE culture
Interactions between
Academia and EC & MS
Conference
COMPIE –
Counterfactual Methods for
Policy Impact Evaluation
Aim: to cross-breed theoretical and applied research on counterfactual methods, with emphasis on ESF CIEs
Where: Rome (Italian Semester)
Dates: November 27-28, 2014

21. For more information http://crie.jrc.ec.europa.eu/COMPIE
Support to MS: promotion of CIE
For more information

22. For more information and contacts: http://crie.jrc.ec.europa.eu

23. For more information and contacts: http://crie.jrc.ec.europa.eu