Counterfactual Impact Evaluation in the Framework of ESF 13/5/2015 Rome Evaluating impacts of hiring incentives in Italy Jeannette Monier Impact Assessment and Evaluation Unit

ESF evaluations during 2007-2013 By December 2013, the ESF Expert Evaluation Network had identified 721 evaluations (finalised and on-going) Most evaluations are conducted at national or multiregional level Most were process evaluations Counterfactual impact evaluations make up about 5% of the total number of evaluations identified

More impact evaluations required In this context, there is a need to supplement existing evaluation practice with approaches that generate much stronger evidence of the impacts of interventions: One approach to capture impacts is through counterfactual evaluations Theory-based evaluation methods may be required in addition to identify the underlying causal mechanisms

DG EMPL initiatives related to CIE Organisation of learning events on CIEs Publication of a Guide for ESF Managing Authorities based on contributions from Member States and evaluation experts http://ec.europa.eu/social/BlobServlet?docId=10805&langId=en Two rounds of Pilot Projects to carry out ESF related CIEs Centre for Research on Impact Evaluation (CRIE)

CRIE CRIE work program consists of three blocks: Support to Member States Support to DG EMPL Dissemination activities http://crie.jrc.ec.europa.eu

Regulatory requirements for 2014-2020 Results-orientation and thematic concentration make it necessary to design programmes that focus their resources on a few objectives in order to maximise their impact. Member States evaluations should help improve the quality of the design and implementation of programmes and also assess their effectiveness, efficiency and impact.

What impacts? Contribution to the objectives of each PA (Art. 56.3) Impact in relation to the EU2020 targets (Art.54) What does this mean? Each SO should be covered The extent will differ according to the duration and size of the intervention, the nature of the result, available datasets, policy importance EU 2020: in most cases, qualitative analysis

ESF specific requirements Explicit requirement introduced for the MS to record and store microdata (on participants). Two evaluations of the YEI are required, one by end-2015, that will inform the overall process review of YEI and will feed into the mid-term budgetary review of the MFF, and one by end-2018. In order to capture impacts of YEI interventions, DG EMPL is investigating possible avenues to apply counterfactual methods in this area

Evaluation Plans Evaluation Plan submitted to MC no later than a year after adoption of OP: Art. 56,1 of CPR MC approves, reviews, amends: Art. 114, 110 Follow up given to evaluation findings: Art.110 From the Evaluation Plans received so far by DG EMPL, one conclusion to be drawn is that there is an increased number of counterfactual studies

Thank you for your attention! Jeannette.Monier@ec.europa.eu