Veronica Gaffey & Antonella Schulte-Braucks Results and Performance for Cohesion Policy A Step Change for the Future Veronica Gaffey & Antonella Schulte-Braucks DG REGIO / DG EMPL Vilnius, 4 July 2013 Cohesion Policy

Effectiveness an overall principle Commission and Member States shall ensure the effectiveness of the ESI Funds, in particular through monitoring, reporting and evaluation (Art.4 CPR) Cohesion Policy 2

Four building blocks for result orientation Result orientation of programmes Indicators Performance framework, review and reserve Evaluation and Reporting Objective is the same but technical details vary by Fund – due to the scope of the Funds Cohesion Policy

What is a result? = what each priority axis and investment priority wants to change / achieve for each category of region or target group E.g., accessibility of a region, increase the number of start-ups of young people below 25. The measurable dimension of results must be captured with indicators: E.g. Reduction in travelling time Number of start-ups of young people. Cohesion Policy 4

Common Indicators Included in annexes to the fund specific Regulations (indicator title and measurement unit) Definitions in Guidances, developed in consultation with the Member States Special attention to measurement units and reporting conventions 5

Investment priorities Result indicators Specific objectives what the MS wants to achieve, taking into account national and regional challenges, needs and potentials Investment priorities Description of actions needed to achieve the specific objectives Output Indicators Categories of intervention Cohesion Policy

Performance Framework Building Blocks Priority level Milestones (intermediate targets) Targets To be achieved by the end of 2018 Formally reviewed in 2019 To be achieved by the end of 2022 (2023) Formally reviewed in 2024 Indicators: Financial, Output, Result (?), Key implementation steps Possible postponement of the deadline due to N+3 rule. Realistic and achievable (taking into account – inter alia – the timeframe, form of intervention and resources) Relevant to the objective of the priority Ex-ante evaluation may help to judge. Revision possible (art. 26) in duly justified cases, such as a significant change in the economic, environmental and labour market conditions in a Member State or region, and in addition to amendments resulting from changes in allocations for a given priority. Cohesion Policy

Challenges for Performance Frameworks Keeping it as simple as possible (limited number of indicators) Coverage of the Priority Axis Setting realistic milestones and targets Consistency within a MS for similar priorities (should be described in Partnership Agreement) Cohesion Policy 8

Performance review & Allocation of Reserve In 2019, examines the performance of the programme priorities against the milestones set for the end of 2018 the European Commission adopts a decision on priorities which have attained their milestones MS proposes the attribution of the performance reserve (7%) among the performing priorities. Foreseen success rate: all indicators reach 85% of a milestone. Where a Member State fails to submit the information in accordance with Article 44(4) and (5), the performance reserve for the programmes or the priorities concerned shall not be allocated to the programmes concerned] Cohesion Policy

Suspensions or corrections EC may suspend all or part of an interim payment of a priority if: serious failure to achieve the milestones due to clearly identified implementation weakness. EC has communicated previously this clearly identified implementation weaknesses and the MS has failed to take the necessary corrective action. At the end of programming period, the EC may apply financial corrections if, in addition: no significant socioeconomic, environmental developments or implementation delays beyond control of the MS prevented the achievement of targets. What a serious failure is? At least two indicators below 65%. Cohesion Policy

Evaluation Ex Ante: Ongoing: Focused more strongly on intervention logic Possibility to use ex ante to obtain data for baselines Assessment of performance framework Ongoing: Evaluation Plan – to MC no later than one year after programme: timing, evaluations, data, methods, communication/use, budget Impact evaluation(s) covering each priority Report summarising evidence and main outputs and results of the programme in December 2021 Cohesion Policy Regional Policy 11

Impact Evaluation No one method favoured over any other All evaluations should pay attention to the theory of change, and mobilise an appropriate mix of methods to conclude on the effects of interventions, including: Literature review, including economic theory, previous evaluation results, etc. Review of administrative and monitoring data – on beneficiaries/ participants/entities Quantitative counterfactual work where appropriate Qualitative techniques: interviews, focus groups, case studies, performance story reporting, etc., etc Cohesion Policy Regional Policy 12

Reporting Annual report by Commission to Council and Parliament, summarising AIR data (outputs, results, performance framework progress) and all available evaluations of Programmes General Affairs Council to discuss every 2 years Quality of Reporting – quantitative and qualitative – must improve for all MS and all Funds Cohesion Policy Regional Policy 13

ERDF/CF Specific Requirements New Focus on Results: What do you want to change? What indicator can capture this change? What is the baseline (the situation before the programme)? How will the outputs of the programme contribute to change? Results relate to change in the region/sector – not just for supported entities

Role of Result Indicators for ERDF/CF Capture what you want to change Should be sufficiently close to policy – policy to be reflected in the evolution of the result indicator Targets quantitative or qualitative Regular monitoring to prompt policy debate (not sanctions) Selected by programmes – not common– recognising the different "journeys" to EU2020 Evaluation to disentangle the contribution of the policy to change from the influence of other factors (impact) Possibility to evaluate impact because the objective was clear 16

ERDF/CF Output Indicators: Common & Specific Capture what the resources are spent on Common & Programme Specific Baselines zero, Quantified Cumulative Targets Intervention logic - how should this amount of resources spent on these outputs contribute to change in result indicator – to be assessed in ex ante evaluation Common Indicators: Relate to most frequently implemented actions Provide aggregate information for analysis & communication purposes 17

ESF Specific requirements Focus on results: What do you want to change? What indicator can capture this change? What is the baseline? How will the outputs of the programme contribute to change? Results relate to change for supported persons/entities, not in the region/sector = Short link – in terms of causality 18

19

ESF Common Result Indicators: Immediate results: change upon leaving the intervention Longer term results: change after e.g. 6 months 20

ESF common output and result indicators IPs shall always report against all common indicators Limited number of indicators linked with a target Baselines for indicators with a target Baselines could be set using experience from previous programmes Cumulative targets, quantified for common indicators, quantified or qualified for programme- specific 21

Recommendations Draft OPs with a clear intervention logic = prerequisite for evaluability Prepare comprehensive evaluation plans which will allow to assess the effects of the interventions Set up appropriate data collection arrangements which will deliver good quality and timely data 22

Guidance documents ERDF + Cohesion Fund ESF http://ec.europa.eu/regional_policy/information/evaluations/guidance_en.cfm#2 ESF http://portal.empl.cec/Management/evaluation/evaluation/esf/Documents/Forms/AllItems.aspx Ex ante evaluation (ERDF, ESF, CF) Performance framework (ERDF, ESF, CF, EAFRD, EMFF) To follow… Cohesion Policy