1. Counterfactual Impact Evaluation in the Framework of ESF
13/5/2015 Rome
Evaluating impacts of hiring incentives in Italy
Jeannette Monier
Impact Assessment and Evaluation Unit

2. ESF evaluations during 2007-2013
By December 2013, the ESF Expert Evaluation Network had identified 721 evaluations (finalised and on-going)
Most evaluations are conducted at national or multiregional level
Most were process evaluations
Counterfactual impact evaluations make up about 5% of the total number of evaluations identified

3. More impact evaluations required
In this context, there is a need to supplement existing evaluation practice with approaches that generate much stronger evidence of the impacts of interventions:
One approach to capture impacts is through counterfactual evaluations
Theory-based evaluation methods may be required in addition to identify the underlying causal mechanisms

4. DG EMPL initiatives related to CIE
Organisation of learning events on CIEs
Publication of a Guide for ESF Managing Authorities based on contributions from Member States and evaluation experts
Two rounds of Pilot Projects to carry out ESF related CIEs
Centre for Research on Impact Evaluation (CRIE)

5. CRIE CRIE work program consists of three blocks:
Support to Member States
Support to DG EMPL
Dissemination activities

6. Regulatory requirements for 2014-2020
Results-orientation and thematic concentration make it necessary to design programmes that focus their resources on a few objectives in order to maximise their impact.
Member States evaluations should help improve the quality of the design and implementation of programmes and also assess their effectiveness, efficiency and impact.

7. What impacts? Contribution to the objectives of each PA (Art. 56.3)
Impact in relation to the EU2020 targets (Art.54)
What does this mean?
Each SO should be covered
The extent will differ according to the duration and size of the intervention, the nature of the result, available datasets, policy importance
EU 2020: in most cases, qualitative analysis

8. ESF specific requirements
Explicit requirement introduced for the MS to record and store microdata (on participants).
Two evaluations of the YEI are required, one by end-2015, that will inform the overall process review of YEI and will feed into the mid-term budgetary review of the MFF, and one by end-2018.
In order to capture impacts of YEI interventions, DG EMPL is investigating possible avenues to apply counterfactual methods in this area

9. Evaluation Plans
Evaluation Plan submitted to MC no later than a year after adoption of OP: Art. 56,1 of CPR
MC approves, reviews, amends: Art. 114, 110
Follow up given to evaluation findings: Art.110
From the Evaluation Plans received so far by DG EMPL, one conclusion to be drawn is that there is an increased number of counterfactual studies

10. Thank you for your attention! Jeannette.Monier@ec.europa.eu