1. Commission guidance and support on (counterfactual) impact evaluation of ESF interventions
Frank Siebern-Thomas Georgeta-Alina Ungureanu DG EMPL, Unit A3 Impact Assessment and Evaluation

2. © www.shutterstock.com, ollyy
Commission guidance and support on (counterfactual) impact evaluation of ESF interventions
© ollyy
Georgeta-Alina Ungureanu Frank Siebern-Thomas
DG EMPL, Unit A3, Impact Assessment and Evaluation

3. General context Current programmes are often just designed to spend
Objectives vague
How to recognize success or failure often not clear
Indicators do not express objectives & reporting is not always reliable (and is hardly ever debated)
Most evaluations focus on bottlenecks to implementation – not effects
Consequently, it has been difficult to demonstrate the impact of cohesion policy

4. Evaluation approaches 2007-2013 (source: ESF Evaluation Network)

5. Effectiveness and result orientation
Commission and Member States shall ensure the effectiveness of the ESI Funds, in particular through monitoring, reporting and evaluation (Art.4 CPR)
Member States evaluations shall be carried out to improve the quality of the design and implementation of programmes, as well as to assess their effectiveness, efficiency and impact (Art. 47 (1) CPR)

6. Four building blocks for result orientation
OPs: clear intervention logic based on country specific recommendations
Indicators, especially common indicators
Performance framework, review and reserve
Evaluation and Reporting

7. Evaluation Ex Ante: Ongoing:
Focused more strongly on intervention logic
Possibility to use ex ante to obtain data for baselines
Assessment of performance framework
Ongoing:
Evaluation Plan – to MC no later than one year after programme: timing, evaluations, data, methods, communication/use, budget
Impact evaluation(s) covering each priority
For YEI: two evaluations required assessing i.a. quality of offers, sustainability of employment, quality traineeship. Deadline for evaluations end 2015 and end 2019
Report summarising evidence and main outputs and results of the programme in December 2022

8. Impact Evaluation No one method favoured over any other
All evaluations should pay attention to the theory of change, and mobilise an appropriate mix of methods to conclude on the effects of interventions, including:
Literature review, including economic theory, previous evaluation results, etc.
Review of administrative and monitoring data – on beneficiaries/ participants/entities
Quantitative counterfactual work where appropriate
Qualitative techniques: interviews, focus groups, case studies, performance story reporting, etc.

9. What are counterfactual impact evaluations?
Impacts can be evaluated through various methods, i.a. counterfactual impact evaluations (CIEs)
Impacts (net effects) - the difference between what has happened in the presence of an intervention and what would have happened without
Family of evaluation methods that:
Seek to determine whether an intervention is the cause of change in results
Identify net effects of interventions
Rule out alternative explanations

10. DG EMPL Practical Guidance on CIEs
Team of external experts:
Stephen Morris, National Center for Social Research and the Policy Studies Institute, London
Herta Tödtling-Schönhofer, Metis, Vienna
Michael Wiseman, the George Washington Institute of Public Policy, Washington
The guidance covers:
main concepts and methods (what are CIEs, why CIEs matter, technical challenges, link with other evaluation approaches)
practical considerations in preparing a CIE
suggestions for moving the CIE agenda forward

11. Experimental approach

12. Propensity Score Matching

13. Difference in difference

14. Regression Discontinuity Design

15. Instrumental variables

16. Examples - overview ESF/SSFF Non-ESF/SSFF ?
Random
Assignment ?
UK Employment Retention and Advancement (ERA) Demonstration
(Walker et al, 2006)
Effect of delegation to private providers of placement services for young unemployed graduates in France
(Crepon et al, 2011)
Difference-in-Differences
Evaluation of social integration services for socially vulnerable and socially excluded individuals (ESF Lithuania)
Evaluating the employment impact of a mandatory job search program
– New Deal for Young People
(Blundell et al, 2004)
Impact of smoking free area legislation on smoking behavior in Germany
(Anger et al, 2010)
Propensity Score Matching
Early Impacts of the European Social Fund (ESF England)
Effectiveness of individual voucher (‘dowry’) for training and employment (ESF Italy-Lombardy)
Effects of marginal/subsidised employment on subsequent labour market outcomes (Böheim and Weber, 2011)
Regression Discontinuity Design
Impact of structural funds on economic growth (Busillo et al, 2010)
Impact of extended benefits on unemployment duration in
Austria (Lalive, 2008)
Instrumental Variables
Evaluating OP HRE using the identifiers of grant appraisal experts as an instrument (ESF Czech Republic)
Access to Early Retirement and Mortality (Kuhn et al, 2010, IZA)
Effect of education on voting behaviors in Europe (D’hombres et al, 2010)

17. CIEs and other approaches
CIEs describe the net effects of interventions
Do less well in addressing other questions
Explaining results and how they came about
Nature of the intervention & how it operated in practice
Evaluation design should comprise a range of approaches alongside CIE
Theory-based approaches address questions of impact

18. Selecting interventions for CIEs
not all ESF interventions can be evaluated using CIEs; need for prioritisation and strategic choice
Strategic issues: justify funds, focus of reform process, learning
Nature of intervention ('amenable to CIE'?): Discrete, distinctive and homogenous interventions? Identifiable causal mechanism? Can results be defined quantitatively? Well-defined target group? Sufficient size of treatment group? Interference with other instruments? Meaningful control group?
Data availability: types of data and data protection.

19. Practical considerations in preparing a CIE – planning ahead
Evaluation plan  evaluation process
 activities
 timing, budget
Evaluation scheme  detailed plan
 taylored to specific circumstances and needs that CIEs address
All evaluations
CIE
CIE

20. Data availability Types of data Data protection
Treatment and control group records
Result records
Contextual data/control variables
Data protection
Microdata available?
National data on individual careers to compare ESF participants with control group?
Anonymized data Possibility to de-anonymize data to follow individual careers?

21. Evaluation scheme Objectives, strategy, intervention logic Resources
Timing
Identify target group
Identify control group
Data issues
Reporting and dissemination

22. Center for Research on Impact Evaluation in the JRC in Ispra, Italy
CRIE support to DG EMPL
Pilot projects on counterfactual impact evaluation
European Progress Microfinance Facility
Social policy experimentation
Building bridges between EU policy-makers and the scientific community
Identifying knowledge gaps, i.e. areas in which impact evaluations would be particularly needed, and related production of policy briefs

23. CRIE support to MS
CRIE supports MS to set up the necessary arrangements for carrying out Counterfactual Impact Evaluations of DG EMPL funded interventions by:
Organizing training workshops on impact evaluation methods
Providing tailor-made advice on methodological and data issues that arise when designing, implementing and evaluating an initiative
Provide support to MS for arranging CIEs (preparation and implementation phase)

24. Pilot Projects to carry out ESF related Counterfactual Impact Evaluations
To provide robust evidence on the net effects (impacts) of the ESF interventions being evaluated through the chosen pilot projects
To further raise awareness on counterfactual impact evaluations and on the use of their findings
To enhance MS capacity to effectively implement CIEs

25. First call for proposals
Launched on 17 April 2013
Dead-line for applications: 21 June 2013
Total amount allocated: €1 million
3 projects selected by the Evaluation Committee: from Italy, Hungary and Spain
First pre-financing granted to the three beneficiaries; implementation has started

26. Second call for proposals
A survey was sent to all members of ESF Evaluation Partnership on 25 June 2013
The questions focused on:
main reasons for not applying in the first call
interest in applying if a similar call launched
if yes, on what particular ESF interventions
Replies have been received from 18 MS
Second call was launched on 7 October 2013
Total amount allocated: €650,000
Dead-line for applications 29 November

27. Guidance published in EN, FR and DE
The EN version is currently under:
The FR version
The DE version

28. Thank you for your attention!