Centre for Research on Impact Evaluation CIE in Structural Funds evaluation: challenges and opportunities The CRIE experience and perspective Centre for Research on Impact Evaluation (CRIE, DG-JRC) Paolo Paruolo Joint ESF Evaluation Partnership and ERDF Evaluation Network, Athens 15 May 2014

CIE challenges & opportunities CRIE experience & perspective Outline CIE challenges & opportunities CRIE experience & perspective  important reference: DG EMPL 2013 Guidance Document “Design and commissioning of counterfactual impact evaluation”

CIE challenges: Feasibility issues Is the intervention feasible for CIE? Operational and methodological aspects: Intervention features: Innovative and policy relevant? Distinctive and homogeneous? Theory of change? CIE characteristics: Cost-effective? Value-added? Target group/population? Control/comparison group/population? Eligibility vs. participation (mandatory? randomization?)

CIE challenges: Data collection Data availability dictates if/how a CIE can be implemented: Sample selection and Data collection: Plan in advance: treat. status, outcome, control variables. Control group information crucial  possibly gather it before program starts. Microdata: single eligible units level vs. aggregated data. Data sources: Monitoring data is insufficient Administrative data: contact statistical/governmental offices  merge alternative data sources. Survey data: pre/post intervention surveys

CIE challenges: Implemention Starting point: policy questions and hypotheses to be tested. Evaluation design operationalization: Evaluation design: define comparison group; rules of programme operation  (but not ):CIE method. Ethics requirements. Evaluation team: partnerships (policy makers + privates/public agencies). Evaluation timing and budget definition  data collection. Outsourcing vs. in-house CIEs: can outsource any of sample definition, data collection and CIE implementation but crucial to develop in-house CIE knowledge. CRIE contributes to this build-up of capacity

CRIE’s experience CRIE: Centre for Research on Impact Evaluation Joint DG EMPL-DG JRC initiative Established in June 2013 Support to MS and DG EMPL to set up the necessary arrangements for carrying out Counterfactual Impact Evaluations (CIE) of ESF funded interventions Scientists with a background in Economics and Statistics

CRIE Activities 1. Support to DG EMPL 2. Support to MS 3. Research activities

Support to DG EMPL Knowledge gaps from the academic literature Intervention type 1. Training 2. Employment Incentives 3. Labor Market Services (LMS) MS implementing CIEs* MS-level Knowledge gaps** MS implementing CIEs MS-level Knowledge gaps Target group Unemployed AT(1), DE(21), DK(1), IE(1), LV(1), PL(3), RO(1), SE(4), SI(1) 19 DE(10), IT(2), SE(1), FI(1) 24 DK(3), RO(1), IE(1), NL(1), SE(1), DE(1), AT(1) 20 Young unemployed FR(1), SE(1), UK(2) DE(2), FR(1), IT(1), PL(1), UK(1), SE(1) 21 FR(1), PT(1), SE(1), UK(2) 23 Disadvantaged young unemployed DE(1), FR(1) 25 DE(1) 26 - 27 Old unemployed PT(1) Long-term unemployed DE(6), BG(1), SE(2), PL(1) FR(1), SE(1), UK(1) Low-skilled unemployed DK(1) Employed FR(1) FI(1), ES(1), PL(1), DE(1) Inactive IT(1) Disabled * country for which there is CIEs evidence (number of papers) ** number of country-level knowledge gaps (i.e. # countries not implementing CIEs)

Knowledge gaps from the academic literature 1. Training 23 papers for DE interventions; 5 for SE; newer MS severely under-represented. 34 papers evaluating interventions aimed at unemployed. Disadvantaged young unemployed, long-term unemployed, low-skilled unemployed, employed, disabled categories have 1 to 3 entries → lack of evidence. 2. Employment incentives Majority of CIE evaluations in DE and targeting unemployed, long term unemployed and young unemployed. Lack of evidence for Southern and Eastern Europe. Outcome. P(e), Job stability, earnings 3 months after completing : short term effect Eligibility: registered in PEO, below 30 and specific educational levels Insturment : age (fuzzy rdd- though not clear if fuzzy) Other iv: change in amount of grant- duration of internship (likely effect on outcome)

Knowledge gaps from the academic literature 3. Labour Market Services Less CIEs as compared to Training and Employment incentives. Unemployed again the most frequent target group; young unemployed also high. No CIE evidence for inactive and vulnerable groups. No concentration of interventions in some particular EU countries. Highest CIE numbers for DK and UK. With the exception of RO, no evidence for east European MS. Outcome. P(e), Job stability, earnings 3 months after completing : short term effect Eligibility: registered in PEO, below 30 and specific educational levels Insturment : age (fuzzy rdd- though not clear if fuzzy) Other iv: change in amount of grant- duration of internship (likely effect on outcome)

CRIE’s perspective More CIEs are needed They are feasible, especially within the new 2014-2020 programming period

Support to MS: CIE training workshops Mainly two types: 1 day or 2-3 day modules Member State Date Training type Poland Nov. 2013 2 days Spain Dec. 2013 1/2 day intro. Hungary Jan. 2014 1 day intro. Czech Republic Bulgaria Apr. 2014 3 days Estonia Slovak Republic 21-23 May Latvia 3-5 June Lithuania To be decided Belgium Past and future trainings

CIE training workshops Support to MS: CIE training workshops Example of a 2-day training workshop Schedule: Day1 Introduction – Randomized control trials, with examples Propensity score matching method, with examples Difference in difference method, with examples Group exercise Day2 Regression discontinuity design, with examples Instrumental variables and combination of methods, with examples Practical considerations in preparing a CIE

Trainings: satisfaction of participants:

Trainings: satisfaction of participants

Participants’ suggestions: Longer course (requested also by 3-day training participants) More real-world examples and exercises and more details on practical/methodological issues. Computer software course on CIE methods

CRIE’s perspective on Training Workshops Need for a continued support to MS over the 2014-2020 programming period. Future evolution: Training workshops Computer-based Training workshops Workshops with training tutorials Workshops with discussions (at various geographical levels).

Support to MS: direct support Possible data analysis collaboration Tailor-made advice on methodological and data issues that will arise while implementing the CIE Example: Pilot kick-off meetings, Social Experimentation kick-off meetings, … Collaboration with the quantitative analysis Example: Latvia Finance Ministry delegation (March 2014): 2-day presentation in Ispra Possible data analysis collaboration

CRIE’s perspective on collaborations with MS Evaluation of past intervention is difficult because retrieving the right data is not so obvious: need to plan ahead Merging data sets is the only available option for the 2007-2013; this need not be the case in the 2014-2020 programming period, when you have more options

promotion of a CIE culture Support to MS: promotion of a CIE culture Interactions between Academia and EC & MS Conference COMPIE – Counterfactual Methods for Policy Impact Evaluation Aim: to cross-breed theoretical and applied research on counterfactual methods, with emphasis on ESF CIEs Where: Rome (Italian Semester) Dates: November 27-28, 2014

For more information http://crie.jrc.ec.europa.eu/COMPIE Support to MS: promotion of CIE For more information http://crie.jrc.ec.europa.eu/COMPIE

For more information and contacts: http://crie.jrc.ec.europa.eu

For more information and contacts: http://crie.jrc.ec.europa.eu