Ex post evaluation of Cohesion Policy programmes , focusing on the European Regional Development Fund (ERDF) and Cohesion Fund (CF) – Work Package Zero: Data Collection and quality assessment Evaluation Network Meeting Prepared by: Giovanni Familiari - t33 EC- DG Regional and Urban Policy Brussels, 15 January 2015

Overview of the tasks Task 2 Task 2 gather and discuss with MAs  indicator data reported by the MAs in 2012 and 2013 AIRs and in SFC; and  assesses their quality (definitions and reliability). Task 1 Task 1 examines reporting on the use of the Funds  content and reliability;  existing issues with breaking down at NUTS 3;  availability of NUTS 3 expenditure data by priority theme. Task 3 Task 3 collects and aggregates major project data in 2013 AIRs; (Task 4 (Task 4 identifies good practice in monitoring greenhouse gas reduction (2 slides in annex - but not presented))

Overall methodological approach Activities Task 1 Task 2 Task 3 Task 4 Design of tools √√√√ Desk review of AIRs 2012 and 2013 √√√ Interviews and enquiries with MAs and national coordination (maximum coverage) √√√√ Quality control and aggregation of the data √√√√ Reporting √√√√ Quality control and finalisation of reporting √√√√

Task 2: Core indicators (use) Findings: -> all 21 selected indicators used -> Jobs created (no.1) by far the most used, 26 MS ( OPs) -> Cooperation projects enterprises- research institutions and start-ups supported broadly used -> Infrastructure indicators less frequent

Task 2: Core indicators (achievement) Tangible results delivered, e.g.: ≈ jobs created ≈ research jobs > jobs in SMEs ≈ start-ups supported ≈5 million more people with broadband access ≈4.2 million more people now served by water supply

Task 2: Core indicators (MA changes to 2012 data) Number of selected core indicators with achievement values 2012 modified in AIRs 2013: 168 Number of involved MS: 21 Highest number of corrections: DE (33), IT (30) and PL (21); other MS <12 Core indicators involved: 1 ‘jobs created’ (28), 29 ‘Area rehabilitated (km2)’ (17) Reasons: (1) quality control by the MAs; (2) modified methodologies (3) revision of OPs; (4) modified reporting from recipients; (5) corrections allowing consistency with EC recommendations; (6) misinterpretation or misunderstanding of WP0 national experts.

Task 2: Core indicators (WP0 corrected values) Number of selected core indicators with corrected achievement values (>0): 78 (17 MS) Highest number of corrections: CZ (21), DE (13) and PL (10); other MS <7 Main core indicator involved: 29 ‘rehabilitated area’ (40), hectares instead of km2. Other unit of measurement issues: kilowatt instead of megawatt, million people instead of persons. Other reasons for corrections: typos in indicator values, rounding issues. In some cases, the MAs confirmed to consider the changes in the future. In others they left it open.

Task 2: Core indicators (inconsistency with WD7) Number of selected core indicators regarded as not consistent with WD7 (=0): 69 (18 MS) Highest number of inconsistencies: PL (14), FR (13); other MS <8 Main core indicator involved: 24 ‘Additional capacity of renewable energy production’ (22), wrong measurement unit (often ‘MWh’ or ‘KWh’ instead of the correct ‘MW’). 22 ‘Additional population served with improved urban transport’ (18), mostly PL, the indicator refers to ‘passenger rides’ and not the ‘additional population served’. 12 ‘Number of additional population covered by broadband access’ (8), e.g. number of businesses, number of households or various share numbers.

Task 2: Core indicators (inconsistency with WD7) Main reasons for indicator inconsistency: 1.Different measurement object, e.g. o number of enterprises benefiting from flood protection measures; o additional households/enterprises instead of population covered by broadband; o net instead of gross jobs created; o all cooperation projects, not only with research institutions, or number of enterprises and institutions involved in cooperation projects; 2.Data referring to the situation in the programme area and not programme-related, e.g. regional capacity of renewable energy production, jobs in tourism sector MAs confirmed that indicators are not consistent with the EC recommendation. However, they do not dispose of appropriate methods to estimate an achievement figure consistent with EC recommendations in most cases.

Task 2: Other significant OP indicators  Programme specific indicators cover each ex-post evaluation theme  Aggregation not possible to establish reliable information at European level  Such an aggregation would risk missing out on the achievements of programmes which did not set up a corresponding indicator  Other indicators might be used to exemplify specific aspects of the relevant theme

Task 2: Common indicators Results include: ≈ enterprises supported ≈ enterprises cooperating with research institutions ≈7.0 million people covered by improved health services ≈29 million t/y in additional waste recycling capacity

Q&A on Indicators Questions on the work on indicators?

Task 1 – Identifying problem areas for NUTS 3 level regional breakdown of expenditure Findings: -> ≈30% allocated at NUTS3 -> 9 MS do not provide any NUTS3 information Large variations between: -> type, from ETC (2%) to regional (38%) OPs -> size, from small (16%) to large (31%) OPs

Task 1 – Identifying problem areas for NUTS 3 level regional breakdown of expenditure Large variations between: -> priority theme, from ‘IT services and infrastructure’ (13%) to ‘Social infrastructure’ (50%) * size of the bubble represents the share of the priority theme out of the total amounts allocated Allocation to priority themes: share of NUTS3 location*

Task 1 – Identifying problem areas for NUTS 3 level regional breakdown of expenditure Existing issues with breaking down at NUTS 3: -> Non-technical issues  No tier of administration at NUTS3  NUTS3 not relevant to programme / operation objectives -> Data issues  Location data (recorded or not at NUTS3) does not allow to split allocated amounts in a relevant way -> Other  Territories corresponding to different levels of classification In general, NUTS3 detail more often available than reported

Task 1 – Identifying problem areas for NUTS 3 level regional breakdown of expenditure Content and reliability of location data:  Encoded at grant decision, directly into management and information systems  Sometimes not referring to location of final recipients or the scope of project activities  Not always clear when it is estimated  MAs generally aware of differences between decided and allocated amounts  Using expenditure instead of allocation data perceived as not creating additional problems

Task 3: Major projects  Contribution to financial allocation and expenditure reporting incomplete and not sufficiently consistent across the MS.  Contribution to core indicator reporting provided only in a few cases.  More generally, type of information and level of detail given in the AIRs varies widely, even within the same MS, e.g. Poland or Romania. The terminology used also differs.  49 approved major projects where physical works have not yet started (≈7% of total approved) -> more than half of these are located in the EU-15. AIR not a reliable source of info on major project implemention

Thank you!

Task 4: GHG emission reduction  75 OPs report on greenhouse gases (GHG) emissions reductions using core indicator 30.  Definition/unit of measurement used is not consistent in 5 cases. The remaining 70 OPs represent 21.9% of all OPs.  AT (89% of OPs), IT (79%), only some OPs in other MS, e.g. UK 25%, and 13 other MS do not report the indicator at all, e.g. Spain.  Sectors: energy efficiency and renewable energy (44 and 43 OPs), sustainable transport (17) and waste management (6 OPs)  Project-based approach most commonly used for achievements  Method for target and achievements not always consistent

Task 4: GHG emission reduction  Consistency with EC recommendations (from 75 to 70 OPs)  Clarity (from 70 to 30 OPs)  Consistent quantification of achievement and target values (from 30 to 12 OPs)  Reliability (12 OPs)  Replicability (10 OPs, 3 good practices): o All AT OPs using CI30 o OP Basse-Normandie o OP Baden-Württemberg