Commission guidance and support on (counterfactual) impact evaluation of ESF interventions Frank Siebern-Thomas Georgeta-Alina Ungureanu DG EMPL, Unit A3 Impact Assessment and Evaluation

© www.shutterstock.com, ollyy Commission guidance and support on (counterfactual) impact evaluation of ESF interventions © www.shutterstock.com, ollyy Georgeta-Alina Ungureanu Frank Siebern-Thomas DG EMPL, Unit A3, Impact Assessment and Evaluation

General context Current programmes are often just designed to spend Objectives vague How to recognize success or failure often not clear Indicators do not express objectives & reporting is not always reliable (and is hardly ever debated) Most evaluations focus on bottlenecks to implementation – not effects Consequently, it has been difficult to demonstrate the impact of cohesion policy

Evaluation approaches 2007-2013 (source: ESF Evaluation Network)

Effectiveness and result orientation Commission and Member States shall ensure the effectiveness of the ESI Funds, in particular through monitoring, reporting and evaluation (Art.4 CPR) Member States evaluations shall be carried out to improve the quality of the design and implementation of programmes, as well as to assess their effectiveness, efficiency and impact (Art. 47 (1) CPR)

Four building blocks for result orientation OPs: clear intervention logic based on country specific recommendations Indicators, especially common indicators Performance framework, review and reserve Evaluation and Reporting

Evaluation Ex Ante: Ongoing: Focused more strongly on intervention logic Possibility to use ex ante to obtain data for baselines Assessment of performance framework Ongoing: Evaluation Plan – to MC no later than one year after programme: timing, evaluations, data, methods, communication/use, budget Impact evaluation(s) covering each priority For YEI: two evaluations required assessing i.a. quality of offers, sustainability of employment, quality traineeship. Deadline for evaluations end 2015 and end 2019 Report summarising evidence and main outputs and results of the programme in December 2022

Impact Evaluation No one method favoured over any other All evaluations should pay attention to the theory of change, and mobilise an appropriate mix of methods to conclude on the effects of interventions, including: Literature review, including economic theory, previous evaluation results, etc. Review of administrative and monitoring data – on beneficiaries/ participants/entities Quantitative counterfactual work where appropriate Qualitative techniques: interviews, focus groups, case studies, performance story reporting, etc.

What are counterfactual impact evaluations? Impacts can be evaluated through various methods, i.a. counterfactual impact evaluations (CIEs) Impacts (net effects) - the difference between what has happened in the presence of an intervention and what would have happened without Family of evaluation methods that: Seek to determine whether an intervention is the cause of change in results Identify net effects of interventions Rule out alternative explanations

DG EMPL Practical Guidance on CIEs Team of external experts: Stephen Morris, National Center for Social Research and the Policy Studies Institute, London Herta Tödtling-Schönhofer, Metis, Vienna Michael Wiseman, the George Washington Institute of Public Policy, Washington The guidance covers: main concepts and methods (what are CIEs, why CIEs matter, technical challenges, link with other evaluation approaches) practical considerations in preparing a CIE suggestions for moving the CIE agenda forward

Experimental approach

Propensity Score Matching

Difference in difference

Regression Discontinuity Design

Instrumental variables

Examples - overview ESF/SSFF Non-ESF/SSFF ? Random Assignment ? UK Employment Retention and Advancement (ERA) Demonstration (Walker et al, 2006) Effect of delegation to private providers of placement services for young unemployed graduates in France (Crepon et al, 2011) Difference-in-Differences Evaluation of social integration services for socially vulnerable and socially excluded individuals (ESF Lithuania) Evaluating the employment impact of a mandatory job search program – New Deal for Young People (Blundell et al, 2004) Impact of smoking free area legislation on smoking behavior in Germany (Anger et al, 2010) Propensity Score Matching Early Impacts of the European Social Fund 2007-13 (ESF England) Effectiveness of individual voucher (‘dowry’) for training and employment (ESF Italy-Lombardy) Effects of marginal/subsidised employment on subsequent labour market outcomes (Böheim and Weber, 2011) Regression Discontinuity Design Impact of structural funds on economic growth (Busillo et al, 2010) Impact of extended benefits on unemployment duration in Austria (Lalive, 2008) Instrumental Variables Evaluating OP HRE using the identifiers of grant appraisal experts as an instrument (ESF Czech Republic) Access to Early Retirement and Mortality (Kuhn et al, 2010, IZA) Effect of education on voting behaviors in Europe (D’hombres et al, 2010)

CIEs and other approaches CIEs describe the net effects of interventions Do less well in addressing other questions Explaining results and how they came about Nature of the intervention & how it operated in practice Evaluation design should comprise a range of approaches alongside CIE Theory-based approaches address questions of impact

Selecting interventions for CIEs not all ESF interventions can be evaluated using CIEs; need for prioritisation and strategic choice Strategic issues: justify funds, focus of reform process, learning Nature of intervention ('amenable to CIE'?): Discrete, distinctive and homogenous interventions? Identifiable causal mechanism? Can results be defined quantitatively? Well-defined target group? Sufficient size of treatment group? Interference with other instruments? Meaningful control group? Data availability: types of data and data protection.

Practical considerations in preparing a CIE – planning ahead Evaluation plan  evaluation process  activities  timing, budget Evaluation scheme  detailed plan  taylored to specific circumstances and needs that CIEs address All evaluations CIE CIE

Data availability Types of data Data protection Treatment and control group records Result records Contextual data/control variables Data protection Microdata available? National data on individual careers to compare ESF participants with control group? Anonymized data Possibility to de-anonymize data to follow individual careers?

Evaluation scheme Objectives, strategy, intervention logic Resources Timing Identify target group Identify control group Data issues Reporting and dissemination

Center for Research on Impact Evaluation in the JRC in Ispra, Italy CRIE support to DG EMPL Pilot projects on counterfactual impact evaluation European Progress Microfinance Facility Social policy experimentation Building bridges between EU policy-makers and the scientific community Identifying knowledge gaps, i.e. areas in which impact evaluations would be particularly needed, and related production of policy briefs

CRIE support to MS CRIE supports MS to set up the necessary arrangements for carrying out Counterfactual Impact Evaluations of DG EMPL funded interventions by: Organizing training workshops on impact evaluation methods Providing tailor-made advice on methodological and data issues that arise when designing, implementing and evaluating an initiative Provide support to MS for arranging CIEs (preparation and implementation phase)

Pilot Projects to carry out ESF related Counterfactual Impact Evaluations To provide robust evidence on the net effects (impacts) of the ESF interventions being evaluated through the chosen pilot projects To further raise awareness on counterfactual impact evaluations and on the use of their findings To enhance MS capacity to effectively implement CIEs

First call for proposals Launched on 17 April 2013 Dead-line for applications: 21 June 2013 Total amount allocated: €1 million 3 projects selected by the Evaluation Committee: from Italy, Hungary and Spain First pre-financing granted to the three beneficiaries; implementation has started

Second call for proposals A survey was sent to all members of ESF Evaluation Partnership on 25 June 2013 The questions focused on: main reasons for not applying in the first call interest in applying if a similar call launched if yes, on what particular ESF interventions Replies have been received from 18 MS Second call was launched on 7 October 2013 Total amount allocated: €650,000 Dead-line for applications 29 November

Guidance published in EN, FR and DE The EN version is currently under: http://ec.europa.eu/social/BlobServlet?docId=10805&langId=en The FR version http://ec.europa.eu/social/BlobServlet?docId=10805&langId=fr The DE version http://ec.europa.eu/social/BlobServlet?docId=10805&langId=de

Thank you for your attention! Georgeta-Alina.Ungureanu@ec.europa.eu Frank.Siebern@ec.europa.eu http://crie.jrc.ec.europa.eu