ESF 2007-2013 Results of Ex-Post Evaluation Impact of the EU investments on Lithuanian economy, people and quality of life, 23 February 2017 Jeannette Monier Deputy Head of Unit, Evaluation and Impact Assessment Unit DG Employment, Social Affairs and Inclusion During this presentation I will present the main results of the ex-post evaluation of the 2007-2013 programming period: the approach, main findings and lessons learned The ex-post evaluation is a requirement set out in article 49.3 of the Council Regulation 1083/2006 (general provisions). Main objectives are: Transparency - review where and how the 2007-2013 funds were spent, their impacts Accountability - determine what worked and what did not, and to what extent Cohesion policy objectives were achieved Improvement - learn what could be improved in future The Commission is responsible for organising the ex-post evaluations, The SWD presenting the results was published on December 12, 2016

24 Clusters of interventions ESF Evaluation 24 Clusters of interventions Thematic studies Synthesis study Preparatory study Human capital Synthesis including: Update 2014 data + 28 country reports SIC and PP Commission Staff Working Document Access to employment Open Public Consultation Social inclusion Design of the evaluation Regarding the organisation of the ESF, the process of the ex-post evaluation started in 2013 with a Preparatory study, which, building on the lessons of previous evaluations and taking account of the known data limitations, advised on the design the evaluation. The approach suggested was to organise the evaluation through thematic studies and collect more detailed lessons by clustering ESF types of interventions. As a result of it 3 thematic studies were launched : Investment in Human capital (also includes adaptability of workers) Supporting the integration of disadvantaged groups Access and sustainable integration into employment Each thematic study attempted to mobilise all existing evidence from monitoring and evaluation available in the MS and supplement it with an in-depth analysis of a selection of interventions in some MS. The selected interventions were grouped in clusters reflecting the variety of interventions of ESF. Each MS was covered by at least one thematic evaluation, A synthesis was launched, bringing together the major findings and lessons from the thematic reports, updating results with monitoring 2014 data, including Croatia and conducting a dedicated analysis of the support to partnerships and institutional capacity building. Finally, a 12 weeks internet-based public consultation was carried out during the first quarter of 2016 to gather further views from stakeholders and citizens.

Evaluation questions Art. 49.3 of Council Regulation 1083/2006 Extent to which the resources were used Effectiveness Efficiency Socio-economic impact Additionnal criteria: Community Added Value Gender sensitivity Sustainability Lessons Learned Above, we mention the criteria required by the General Regulation, but the DG decided to add some criteria to complete the picture. All these criteria taken together respond to the requirements of better regulation (which were adopted when the evaluation was already ongoing) NB: an assessment of gender sensitivity is in line with the gender equality principle enshrined in article 16 of the General Regulation

Key messages The 2007-2013 ESF was implemented in challenging times, the flexibility in existing programmes mitigated negative effects of economic and financial crisis The programmes have been effective in helping people and supporting systems The ESF helped EU cohesion and has been instrumental in supporting EU strategic objectives, national policies and related CSRs The 2007-13 ESF programmes were implemented in a period of deep global economic and financial crisis. This strongly influenced the business opportunities and the private investment climate, especially in poorer regions. It also influenced public finances and the capacity of governments to invest. The flexibility in existing programmes enabled responses to emerging challenges and mitigated negative effects of the severe economic crisis, which especially affected the most vulnerable groups in society. The interventions reached target groups in need of support, integrating people into the labour market, helping them to gain jobs, improving skills and generating changes in systems. The ESF programmes helped EU cohesion. The ESF provided significant financial resources to address employment and social challenges in a majority of MS, enabling an increase in the volume of support. ESF action broadened the existing scope of actions by supporting groups or policy areas that not would otherwise receive support. The ESF also supported innovation that provided role models for mainstreaming at national level and introduced new ways and processes in which various stakeholders work together. The ESF has been instrumental in supporting EU strategic objectives, national employment, skills and social inclusion policies and related CSR.

77 billion EUR of EU budget ESF 2007-2013 77 billion EUR of EU budget Support MS Policies on the basis of the Employment guidelines and Country Specific Recommendations Increasing Adaptability of workers Enhancing Human Capital Social Inclusion of disadvantaged Enhancing Access to Employment Strengthening Institutional Capacity and Promoting Partnerships The ESF during the period 2007-2013 has been the main EU financial instrument in support of the EU objectives for employment, social inclusion and education and training, first under the Lisbon strategy and then under Europe 2020. The EU budget has contributed 77 billion Euro to the ESF programmes, co-financed by 39 billion Euro from the Member States The programmes were strongly aligned to Member States policies and measures. Concentration of ESF investments were intended through supporting the implementation of the country specific recommendations. Background: Volume of the ESF programmes: 115,6 billion Euros, of which 77 billion funded by the EU and the rest by national resources. ESF represents a variable proportion of the total national expenditure in the fields of interventions: higher in the New Member States, "Shared management": programmes were designed and implemented by the Member States themselves, who decide which operations are to be funded and ensure they are soundly implemented .

Where and how ESF 2007-2013 was spent 69% of EU allocation for Convergence regions 45% of total ESF invested in Human Capital priorities (including adaptability), 34% to Access to employment and 14% to Social Inclusion: Proportion of expenditure on human capital higher in Convergence regions (51%) Proportion of expenditure on access to employment and social inclusion higher in RCE areas (58%) ESF was implemented through 117 Operational Programmes. Most of the EU contribution to the ESF was allocated to the programmes in the Convergence Regions, 69%, the less economically advanced areas and with higher unemployment. The analysis of the programmes shows that almost half of the ESF was allocated to the priorities in support of enhancing human capital and the adaptability of workers (45%), followed by support to active labour market measures (34%) and social inclusion (14%). The Proportion of expenditure on human capital higher in Convergence regions (51%) Proportion of expenditure on access to employment and social inclusion higher in RCE areas (58%)

Where and how ESF 2007-2013 was spent By percentage: A2E 34,3% TA 3,1% HC 45,5% P 0,7% SIC: 2,1% SI 14,3% ESF was implemented through 117 Operational Programmes. Most of the EU contribution to the ESF was allocated to the programmes in the Convergence Regions, 69%, the less economically advanced areas and with higher unemployment. The analysis of the programmes shows that almost half of the ESF was allocated to the priorities in support of enhancing human capital and the adaptability of workers (45%), followed by support to active labour market measures (34%) and social inclusion (14%). The Proportion of expenditure on human capital higher in Convergence regions (51%) Proportion of expenditure on access to employment and social inclusion higher in RCE areas (58%)

Key achievements of ESF Financial implementation is on track (79,3% of total allocation at the end of 2014 vs 90,7% in May 2016), while there are considerable differences accross Member States Macro economic simulations show that HC investments (incl. on infrastructure) had positive impacts on GDP (0,25%) and productivity HC investments (incl. on infrastructure) had positive impacts on GDP (025%) and productivity. The estimated effects are much stronger in the central and eastern european countries (1,5 % increase), but they are also positive for the EU-15 (0,2% increase)

Financial execution as of December 2016 Let me know summarise some of the key achievements obtained by ESF until the end of 2014: These resources allowed to fund 98.7 million participations of individuals, The average cost per participation in the SF is about 900 Euros, What has been obtained with these participations? As mentioned above there are limitations to data that certainly underestimate the results. However we can say that as a result of the ESF at least: 9,4 milliion obtained a job 8.7 million also gained a qualification (formal or certificate) 13.7 million obtained other positive result (for instance: acquisition of non-formalised skills or competences, motivation, searching for a job after periods of inactivity) But the ESF, through the support to systems (i.e.: PES, education and training, etc) has also helped over 200 thousand entities to obtain a positive result (public administrations, NGOs, start-ups, etc.) Over 100 thousand products or tools have been developed (such as development of new qualifications, courses, standards, etc) HC investments (incl. on infrastructure) had positive impacts on GDP (025%) and productivity. The estimated effects are much stronger in the central and eastern european countries (1,5 % increase), but they are also positive for the EU-15 (0,2% increase)

Key achievements of ESF-Participations Let me know summarise some of the key achievements obtained by ESF until the end of 2014: These resources allowed to fund 98.7 million participations of individuals, The average cost per participation in the SF is about 900 Euros, What has been obtained with these participations? As mentioned above there are limitations to data that certainly underestimate the results. However we can say that as a result of the ESF at least: 9,4 milliion obtained a job 8.7 million also gained a qualification (formal or certificate) 13.7 million obtained other positive result (for instance: acquisition of non-formalised skills or competences, motivation, searching for a job after periods of inactivity) But the ESF, through the support to systems (i.e.: PES, education and training, etc) has also helped over 200 thousand entities to obtain a positive result (public administrations, NGOs, start-ups, etc.) Over 100 thousand products or tools have been developed (such as development of new qualifications, courses, standards, etc) HC investments (incl. on infrastructure) had positive impacts on GDP (025%) and productivity. The estimated effects are much stronger in the central and eastern european countries (1,5 % increase), but they are also positive for the EU-15 (0,2% increase)

Key achievements of ESF-Results Let me know summarise some of the key achievements obtained by ESF until the end of 2014: These resources allowed to fund 98.7 million participations of individuals, The average cost per participation in the SF is about 900 Euros, What has been obtained with these participations? As mentioned above there are limitations to data that certainly underestimate the results. However we can say that as a result of the ESF at least: 9,4 milliion obtained a job 8.7 million also gained a qualification (formal or certificate) 13.7 million obtained other positive result (for instance: acquisition of non-formalised skills or competences, motivation, searching for a job after periods of inactivity) But the ESF, through the support to systems (i.e.: PES, education and training, etc) has also helped over 200 thousand entities to obtain a positive result (public administrations, NGOs, start-ups, etc.) Over 100 thousand products or tools have been developed (such as development of new qualifications, courses, standards, etc) HC investments (incl. on infrastructure) had positive impacts on GDP (025%) and productivity. The estimated effects are much stronger in the central and eastern european countries (1,5 % increase), but they are also positive for the EU-15 (0,2% increase) Public consultation: positive views on ESF supporting the development of new qualifications, courses, training programmes, standards or systems, rising the competitiveness and adaptability of enterprises, start-ups and improving public administration effectiveness and efficiency

Limitations with data and evaluations Priority Axis in the OP not matching priorities of the ESF Regulation Data problems despite improvements related to Annex XXIII: Data on participations, not participants Sometimes indirect participants reported Disadvantaged groups underreported Incomplete reporting on socio-economic characteristics Data reported only at priority level Lack of common definitions and common results indicators Evaluations from MS providing little evidence on impacts Limitations faced in this evaluation and how these were addressed -design of Programmes: MS were free to formulate the priority axes (Pax) under the programme, the evaluators therefore allocated each Pax to a single policy theme for the purpose of the evaluation, although some tackled several themes - data on participants: although the evaluation could draw for the first time on common indicators related to participations (so called annex xxiii) to aggregate at EU level, the participants data also included indirect participants (to a systems support), and double counting if interventions were split, pointing to an overall overreporting. Disadvantaged groups on the other hand were under-reported, notably due to legal restrictions. Evaluators were therefore prudent in their conclusions. data relating to results: for participants, it was possible to aggregate a fair share of results (corresponding to 70% of the participations or 73% of the allocation) in three categories:gaining or maintaining employment, obtaining a qualification or other positive results (skills, competences…). Also calculation of estimated success rates excluding priorities where no result could be aggregated. 35% of results indicators didn't have a target and 8% were not monitored. In most cases lack of results and targets by gender, and for youth. Soft outcomes, crucial for SI, were rarely measured limitations on evaluations from MS: although the Commission managed to create a database on all evaluations carried out by MS (over 700), these were mainly process related and included very few impact evaluations and even less quantitative ones (so called counterfactual impact evaluations). Those available were used. Also, there was scarce data/evaluations on actions supporting systems. Finally, good practices were difficult to extract due to lack of data at intervention level

Key findings Strong alignment of ESF to EU and national challenges and priorities and matching of OP and CSR Implementation has progressed adequately in terms of absorption and reaching mostly targets set by MS ESF has reached most relevant target groups of participants (low skilled, inactive, youth), but Decreasing relative share of unemployed, migrants, minorities Relative share of youth and older workers remained constant Balanced participation of women (51.4 %), with most programmes applying gender equality as a horizontal principle, but few specific actions and results not measured by gender Evaluators have found a strong alignment of the ESF programmes to the EU and national challenges and priorities. The ESF has actually acted in support to the EU objectives and address the national challenges identified. It was easy to trace the match between the existence of a CSR and measures addressing it in the programmes. By end 2014 an amount of 91.7 billiion Euros had been spent. This makes an absorption rate of 79.3% ; 90,7% mid 2016). Some MS (i.e. RO, SK, MT, HR) or priorities (Institutional capacity building) were slower, but absorption of funds can be considered adequate. Main factors for lagging behind are organisational and lack of capacity The ESF has also reached the most relevant groups in need of support. 45% were low skilled, 36% inactive, 33% employed, 30% young people and another 30% unemployed. It is interesting to note that the relative share of unemployed and some disadvantaged groups decreased over time, while the participation of young people remained unchanged (although the policy attention, such as Youth- action teams, increased) It is important to note the balanced representation of women in the ESF operations: 51% were women.

Key findings ESF interventions generally effective in all policy fields, though comparatively more in individual than in system results (longer time to bear fruit): 63% result targets achieved or exceeded by 2014 30m results/68.9m related participations: 44% results ratio by 2014 Average cost per participant is below 900 € but it hides a wide variety of costs Public consultation points to management and control systems, reporting, and audit as the more burdensome areas, WP 12 of REGIO also flags high burden in project selection and implementation Overal, we can consider that ESF has been generally effective: 55% of the targets set for result indicators were exceeded by end 2014, plus another 8% performed between 90%-100% If we compare the 31,6 million positive results with the participants for which results could be aggregated, the results ratio is 44%, The in-depth analysis of interventions show typically higher success rates The public consultation confirms positive views on performance ESF was most effective in supporting PES and LM institutions; quality of higher education and social inclusion systems Institutional capacity building: success was highly dependent on political support, mutual learning, netwoking and exchange of experience between stakeholders Promotion of Partnerships: success depended on integration to national strategies, proper needs asssessment, and MS capacity to develop concepts and services were key to success The average cost per participant is about 900€, however there is a wide dispartity of costs. This depends on factors such as the policy theme (higher in A2E and SI), role of the ESF (higher for innovative interventions or support to full national implementation of minstream measures), The evaluations did not cover administrative burden issues (ESF was coverd by DG REGIO evaluation on Delivery Systems), but the public consultation did. Result point to management and control systems, reporting and audit as the more burdensome areas, Audit requirements credited for reduction in error rates "single audit" principle not always respected Simplified Cost Options associated with eased burden but limited use in 2007-2013.

Weak evidence on sustainability of results, varies considerably Scarcity of follow-up data on participants measuring sustainability of effects Where it exists, there is considerable variety for individuals, ranging from 20-91 % depending on the nature of intervention and target group Mixed results were found for actions on systems, with the exception of investment in LLL systems Sustainability of systems dependent on conversion of new working methods in lasting networks, sharing of lessons learned and mainstreaming of approaches

ESF 2007-2013 EU value added Volume: additional resources to support EU and national employment policies Help weather the effects of the crisis Scope: Extension of support to new target groups and offering of more tailored services Disabled; Young; Unemployed with low qualifications Role: test and implement innovative activities and reforms to employment and social services, education and training systems Process: support administrative capacity building PES (Public Employment Services) Educational institutions and Healthcare Apart from reaching a signficant number of EU citizens, there are other areas in which ESF has demonstrated EU value added: The ESF has provided additional resources to support national policies. In number of countries ESF finaced majority of ALMP activities (EL, SK, LT). In other ESF covered large proportion of pupils and students at all levels of education (BG, PT), The ESF prevented more negative effects of the crisis and helped MS weather lack of funding during the crisis and maintain investment in education, ALMP and social inclusion The ESF has enabled the extension of support to new target groups and offering more tailored and intensive services to specific groups The ESF has supported innovation and the introduction of reforms in education and training (i.e.: new curricula, apprenticeships, VET reforms), public employment services, and reforms in the public administration (i.e.: reforms in Justice, e-governance and enhancing buisness friendlines of the administration). As a conclusion, the ESF has been an effective instrument to develop a human face of Europe, providing tangible benefits to people and contributing to a more competitive and inclusive Europe,

Lessons learned Policy choices: Programming: Target groups: Continue aligning ESF with EU/national priorities Flexibility to adjust to emerging needs Programming: Robust definition of objectives, targets and results Apply more evidence-based programing Target groups: Ensure coverage of disadvantaged groups Continue focus on young and old and balanced representation by gender Programme Implementation: Promote customisation to the needs of specific target groups Improve capacity building Further simplify procedures and continue reducing administrative burden Monitoring systems: Higher standardisation of programme indicators Improve use of longitudinal and micro-data Capture effects other than employment and qualifications – "soft results" Evaluation: More robust impact evaluations Reintroduce final evaluations/timing of evaluations Continue aligning ESF with EU/national priorities and link with objectives as expressed by CSRs needs to maintained in the forthcoming programming periods Keep flexibilty so that programmes can be adjusted on the face of unexpected emerging needs Deepen the results orientation by strenghtening the definition of objectives, targets and results and applying more evidence-based programming Specific actions need to be taken as to ensure coverage of disadvantaged groups (Roma, migrants). Continue focus on young and old people and balanced representation by gender Improve capacity building at all levels Further simplify and continue reducing administrative burden:. Improve use of longitudinal and micro-data - Capture effects other than employment and qualifications Higher standardisation of programme indicators Review evaluation provisions to foster impact evaluations and reintroduce final evaluations

Changes in the 2014-20 regulations Alignment of ESF with EU priorities – CSRs: thematic objectives in line with Europe 2020 + thematic ex ante conditionalities Enhanced flexibility to adjust programmes Coordination with other ESIF: Partnership agreements + multifund Ops Programmes have to set specific objectives translated into clear indicators of results with targets and benchmarks Regular reporting of results (broken-down by gender) and outputs and a performance framework linked to a performance reserve • for projects up to 50,000 euros

Changes in the 2014-20 regulations Use of SCOs expanded (increased legal certainty and mandatory uptake) YEI integrated into ESF 20% ESF in SI – coverage of disadvantaged Impact evaluation for each of the specific objectives Timing of evaluations: summary report of MS evaluations by December 2022

Lessons requiring attention from COM on 2014-2020 and post 2020 Encourage MS to report on "soft results" Increase uptake of simplification Continue building MS evaluation capacity Better capture the results of capacity building activities of the programme Support and promote use of FEIs Streamlining application of the single audit principle Consider the recommendations of the High Level Group on simplification Other lessons require attention during the 2014-2020 implementation period and post-2020:  ‘Soft’ results were not considered in the common monitoring framework for 2014-2020, but the Commission should encourage Member States to use them, in particular for the purpose of impact evaluations;  The Commission supports Member States’ evaluation capacity on an ongoing basis through its regular meetings with evaluations partners and the ‘community of practice’ by CRIE as well as evaluation helpdesk. In these forums, the Commission fosters the combined use of micro-data and administrative registers to carry out longitudinal studies. It will also seek to address issues such as capturing the effects other than employment and qualifications fostering the use of macro-economic models to estimate impacts;  The Commission will also explore possibilities to better capture the results of capacity building activities, for example by fostering the measurement of empowerment;  The Commission will continue its efforts to persuade Member States to increase the uptake of simplification procedures and reduce administrative burden for beneficiaries, MAs and implementing bodies, such as the recently proposed Delegated Act on SCOs and the simplification of reporting on households in the framework of the ‘Omnibus Regulation’81;  FEI implementation was disappointing and hampered by a slow disbursement of funds and by reporting arrangement deficiencies. Some issues were addressed in 2007-2013 (e.g. through more detailed reporting arrangements) and 2014-2020 periods, but there is a continued need for support and promotion in the use of FEIs for future programming periods;  Although progressive simplification measures were introduced in 2007-2013, the overall level of uptake remained low. The mandatory use of SCOs for small projects in 2014-2020 should be seen as a step in the right direction. Further ways in which the authorities can improve legal certainty should be sought, so as to address the concerns on the part of MAs that prevent the widespread adoption of SCOs;  Although the strengthened audit framework resulted in a reduction of reported error rates as compared with previous programming periods, audit remained one of the areas identified in the open public consultation and the DG REGIO evaluation on delivery systems as excessively burdensome. Means of streamlining the application of the single-audit principle should be considered in forthcoming programming periods; and  The Commission will consider the recommendations of the high-level group of independent experts on monitoring simplification for beneficiaries of the European structural and investment funds. This evaluation will feed into discussions on post-2020 and any impact assessment for future programming periods.

Thank you! For more information on ESF: http://ec.europa.eu/social/main.jsp?catId=701&langId=en http://ec.europa.eu/news/index_en.htm#all|1