1. ESF 2007-2013 Results of Ex-Post Evaluation
Impact of the EU investments on Lithuanian economy, people and quality of life, 23 February 2017
Jeannette Monier
Deputy Head of Unit, Evaluation and Impact Assessment Unit
DG Employment, Social Affairs and Inclusion
During this presentation I will present the main results of the ex-post evaluation of the programming period: the approach, main findings and lessons learned
The ex-post evaluation is a requirement set out in article 49.3 of the Council Regulation 1083/2006 (general provisions). Main objectives are:
Transparency - review where and how the funds were spent, their impacts
Accountability - determine what worked and what did not, and to what extent Cohesion policy objectives were achieved
Improvement - learn what could be improved in future
The Commission is responsible for organising the ex-post evaluations,
The SWD presenting the results was published on December 12, 2016

2. 24 Clusters of interventions
ESF Evaluation
24 Clusters of interventions
Thematic studies
Synthesis study
Preparatory study
Human capital
Synthesis including:
Update 2014 data +
28 country reports
SIC and PP
Commission
Staff
Working
Document
Access to employment
Open Public Consultation
Social inclusion
Design of the evaluation
Regarding the organisation of the ESF, the process of the ex-post evaluation started in 2013 with a Preparatory study, which, building on the lessons of previous evaluations and taking account of the known data limitations, advised on the design the evaluation.
The approach suggested was to organise the evaluation through thematic studies and collect more detailed lessons by clustering ESF types of interventions. As a result of it 3 thematic studies were launched :
Investment in Human capital (also includes adaptability of workers)
Supporting the integration of disadvantaged groups
Access and sustainable integration into employment
Each thematic study attempted to mobilise all existing evidence from monitoring and evaluation available in the MS and supplement it with an in-depth analysis of a selection of interventions in some MS. The selected interventions were grouped in clusters reflecting the variety of interventions of ESF. Each MS was covered by at least one thematic evaluation,
A synthesis was launched, bringing together the major findings and lessons from the thematic reports, updating results with monitoring 2014 data, including Croatia and conducting a dedicated analysis of the support to partnerships and institutional capacity building.
Finally, a 12 weeks internet-based public consultation was carried out during the first quarter of 2016 to gather further views from stakeholders and citizens.

3. Evaluation questions Art. 49.3 of Council Regulation 1083/2006
Extent to which the resources were used
Effectiveness
Efficiency
Socio-economic impact
Additionnal criteria:
Community Added Value
Gender sensitivity
Sustainability
Lessons Learned
Above, we mention the criteria required by the General Regulation, but the DG decided to add some criteria to complete the picture. All these criteria taken together respond to the requirements of better regulation (which were adopted when the evaluation was already ongoing)
NB: an assessment of gender sensitivity is in line with the gender equality principle enshrined in article 16 of the General Regulation

4. Key messages
The ESF was implemented in challenging times, the flexibility in existing programmes mitigated negative effects of economic and financial crisis
The programmes have been effective in helping people and supporting systems
The ESF helped EU cohesion and has been instrumental in supporting EU strategic objectives, national policies and related CSRs
The ESF programmes were implemented in a period of deep global economic and financial crisis. This strongly influenced the business opportunities and the private investment climate, especially in poorer regions. It also influenced public finances and the capacity of governments to invest. The flexibility in existing programmes enabled responses to emerging challenges and mitigated negative effects of the severe economic crisis, which especially affected the most vulnerable groups in society.
The interventions reached target groups in need of support, integrating people into the labour market, helping them to gain jobs, improving skills and generating changes in systems.
The ESF programmes helped EU cohesion. The ESF provided significant financial resources to address employment and social challenges in a majority of MS, enabling an increase in the volume of support. ESF action broadened the existing scope of actions by supporting groups or policy areas that not would otherwise receive support. The ESF also supported innovation that provided role models for mainstreaming at national level and introduced new ways and processes in which various stakeholders work together. The ESF has been instrumental in supporting EU strategic objectives, national employment, skills and social inclusion policies and related CSR.

5. 77 billion EUR of EU budget
ESF
77 billion EUR of EU budget
Support MS Policies on the basis of the Employment guidelines and Country Specific Recommendations
Increasing Adaptability of workers
Enhancing Human Capital
Social Inclusion of disadvantaged
Enhancing Access to Employment
Strengthening Institutional Capacity and
Promoting Partnerships
The ESF during the period has been the main EU financial instrument in support of the EU objectives for employment, social inclusion and education and training, first under the Lisbon strategy and then under Europe 2020.
The EU budget has contributed 77 billion Euro to the ESF programmes, co-financed by 39 billion Euro from the Member States
The programmes were strongly aligned to Member States policies and measures.
Concentration of ESF investments were intended through supporting the implementation of the country specific recommendations.
Background:
Volume of the ESF programmes: 115,6 billion Euros, of which 77 billion funded by the EU and the rest by national resources.
ESF represents a variable proportion of the total national expenditure in the fields of interventions: higher in the New Member States,
"Shared management": programmes were designed and implemented by the Member States themselves, who decide which operations are to be funded and ensure they are soundly implemented .

6. Where and how ESF 2007-2013 was spent
69% of EU allocation for Convergence regions
45% of total ESF invested in Human Capital priorities (including adaptability), 34% to Access to employment and 14% to Social Inclusion:
Proportion of expenditure on human capital higher in Convergence regions (51%)
Proportion of expenditure on access to employment and social inclusion higher in RCE areas (58%)
ESF was implemented through 117 Operational Programmes.
Most of the EU contribution to the ESF was allocated to the programmes in the Convergence Regions, 69%, the less economically advanced areas and with higher unemployment.
The analysis of the programmes shows that almost half of the ESF was allocated to the priorities in support of enhancing human capital and the adaptability of workers (45%), followed by support to active labour market measures (34%) and social inclusion (14%).
The Proportion of expenditure on human capital higher in Convergence regions (51%)
Proportion of expenditure on access to employment and social inclusion higher in RCE areas (58%)

7. Where and how ESF 2007-2013 was spent
By percentage:
A2E 34,3%
TA 3,1%
HC 45,5%
P 0,7%
SIC: 2,1%
SI 14,3%
ESF was implemented through 117 Operational Programmes.
Most of the EU contribution to the ESF was allocated to the programmes in the Convergence Regions, 69%, the less economically advanced areas and with higher unemployment.
The analysis of the programmes shows that almost half of the ESF was allocated to the priorities in support of enhancing human capital and the adaptability of workers (45%), followed by support to active labour market measures (34%) and social inclusion (14%).
The Proportion of expenditure on human capital higher in Convergence regions (51%)
Proportion of expenditure on access to employment and social inclusion higher in RCE areas (58%)

8. Key achievements of ESF
Financial implementation is on track (79,3% of total allocation at the end of 2014 vs 90,7% in May 2016), while there are considerable differences accross Member States
Macro economic simulations show that HC investments (incl. on infrastructure) had positive impacts on GDP (0,25%) and productivity
HC investments (incl. on infrastructure) had positive impacts on GDP (025%) and productivity. The estimated effects are much stronger in the central and eastern european countries (1,5 % increase), but they are also positive for the EU-15 (0,2% increase)

9. Financial execution as of December 2016
Let me know summarise some of the key achievements obtained by ESF until the end of 2014:
These resources allowed to fund 98.7 million participations of individuals,
The average cost per participation in the SF is about 900 Euros,
What has been obtained with these participations? As mentioned above there are limitations to data that certainly underestimate the results. However we can say that as a result of the ESF at least:
9,4 milliion obtained a job
8.7 million also gained a qualification (formal or certificate)
13.7 million obtained other positive result (for instance: acquisition of non-formalised skills or competences, motivation, searching for a job after periods of inactivity)
But the ESF, through the support to systems (i.e.: PES, education and training, etc) has also helped over 200 thousand entities to obtain a positive result (public administrations, NGOs, start-ups, etc.)
Over 100 thousand products or tools have been developed (such as development of new qualifications, courses, standards, etc)
HC investments (incl. on infrastructure) had positive impacts on GDP (025%) and productivity. The estimated effects are much stronger in the central and eastern european countries (1,5 % increase), but they are also positive for the EU-15 (0,2% increase)

10. Key achievements of ESF-Participations
Let me know summarise some of the key achievements obtained by ESF until the end of 2014:
These resources allowed to fund 98.7 million participations of individuals,
The average cost per participation in the SF is about 900 Euros,
What has been obtained with these participations? As mentioned above there are limitations to data that certainly underestimate the results. However we can say that as a result of the ESF at least:
9,4 milliion obtained a job
8.7 million also gained a qualification (formal or certificate)
13.7 million obtained other positive result (for instance: acquisition of non-formalised skills or competences, motivation, searching for a job after periods of inactivity)
But the ESF, through the support to systems (i.e.: PES, education and training, etc) has also helped over 200 thousand entities to obtain a positive result (public administrations, NGOs, start-ups, etc.)
Over 100 thousand products or tools have been developed (such as development of new qualifications, courses, standards, etc)
HC investments (incl. on infrastructure) had positive impacts on GDP (025%) and productivity. The estimated effects are much stronger in the central and eastern european countries (1,5 % increase), but they are also positive for the EU-15 (0,2% increase)

11. Key achievements of ESF-Results
Let me know summarise some of the key achievements obtained by ESF until the end of 2014:
These resources allowed to fund 98.7 million participations of individuals,
The average cost per participation in the SF is about 900 Euros,
What has been obtained with these participations? As mentioned above there are limitations to data that certainly underestimate the results. However we can say that as a result of the ESF at least:
9,4 milliion obtained a job
8.7 million also gained a qualification (formal or certificate)
13.7 million obtained other positive result (for instance: acquisition of non-formalised skills or competences, motivation, searching for a job after periods of inactivity)
But the ESF, through the support to systems (i.e.: PES, education and training, etc) has also helped over 200 thousand entities to obtain a positive result (public administrations, NGOs, start-ups, etc.)
Over 100 thousand products or tools have been developed (such as development of new qualifications, courses, standards, etc)
HC investments (incl. on infrastructure) had positive impacts on GDP (025%) and productivity. The estimated effects are much stronger in the central and eastern european countries (1,5 % increase), but they are also positive for the EU-15 (0,2% increase)
Public consultation: positive views on ESF supporting the development of new qualifications, courses, training programmes, standards or systems, rising the competitiveness and adaptability of enterprises, start-ups and improving public administration effectiveness and efficiency

12. Limitations with data and evaluations
Priority Axis in the OP not matching priorities of the ESF Regulation
Data problems despite improvements related to Annex XXIII:
Data on participations, not participants
Sometimes indirect participants reported
Disadvantaged groups underreported
Incomplete reporting on socio-economic characteristics
Data reported only at priority level
Lack of common definitions and common results indicators
Evaluations from MS providing little evidence on impacts
Limitations faced in this evaluation and how these were addressed
-design of Programmes: MS were free to formulate the priority axes (Pax) under the programme, the evaluators therefore allocated each Pax to a single policy theme for the purpose of the evaluation, although some tackled several themes
- data on participants: although the evaluation could draw for the first time on common indicators related to participations (so called annex xxiii) to aggregate at EU level, the participants data also included indirect participants (to a systems support), and double counting if interventions were split, pointing to an overall overreporting. Disadvantaged groups on the other hand were under-reported, notably due to legal restrictions. Evaluators were therefore prudent in their conclusions.
data relating to results: for participants, it was possible to aggregate a fair share of results (corresponding to 70% of the participations or 73% of the allocation) in three categories:gaining or maintaining employment, obtaining a qualification or other positive results (skills, competences…). Also calculation of estimated success rates excluding priorities where no result could be aggregated. 35% of results indicators didn't have a target and 8% were not monitored. In most cases lack of results and targets by gender, and for youth. Soft outcomes, crucial for SI, were rarely measured
limitations on evaluations from MS: although the Commission managed to create a database on all evaluations carried out by MS (over 700), these were mainly process related and included very few impact evaluations and even less quantitative ones (so called counterfactual impact evaluations). Those available were used. Also, there was scarce data/evaluations on actions supporting systems. Finally, good practices were difficult to extract due to lack of data at intervention level

13. Key findings
Strong alignment of ESF to EU and national challenges and priorities and matching of OP and CSR
Implementation has progressed adequately in terms of absorption and reaching mostly targets set by MS
ESF has reached most relevant target groups of participants (low skilled, inactive, youth), but
Decreasing relative share of unemployed, migrants, minorities
Relative share of youth and older workers remained constant
Balanced participation of women (51.4 %), with most programmes applying gender equality as a horizontal principle, but few specific actions and results not measured by gender
Evaluators have found a strong alignment of the ESF programmes to the EU and national challenges and priorities. The ESF has actually acted in support to the EU objectives and address the national challenges identified. It was easy to trace the match between the existence of a CSR and measures addressing it in the programmes.
By end 2014 an amount of 91.7 billiion Euros had been spent. This makes an absorption rate of 79.3% ; 90,7% mid 2016). Some MS (i.e. RO, SK, MT, HR) or priorities (Institutional capacity building) were slower, but absorption of funds can be considered adequate. Main factors for lagging behind are organisational and lack of capacity
The ESF has also reached the most relevant groups in need of support. 45% were low skilled, 36% inactive, 33% employed, 30% young people and another 30% unemployed.
It is interesting to note that the relative share of unemployed and some disadvantaged groups decreased over time, while the participation of young people remained unchanged (although the policy attention, such as Youth- action teams, increased)
It is important to note the balanced representation of women in the ESF operations: 51% were women.

14. Key findings
ESF interventions generally effective in all policy fields, though comparatively more in individual than in system results (longer time to bear fruit):
63% result targets achieved or exceeded by 2014
30m results/68.9m related participations: 44% results ratio by 2014
Average cost per participant is below 900 € but it hides a wide variety of costs
Public consultation points to management and control systems, reporting, and audit as the more burdensome areas, WP 12 of REGIO also flags high burden in project selection and implementation
Overal, we can consider that ESF has been generally effective:
55% of the targets set for result indicators were exceeded by end 2014, plus another 8% performed between 90%-100%
If we compare the 31,6 million positive results with the participants for which results could be aggregated, the results ratio is 44%,
The in-depth analysis of interventions show typically higher success rates
The public consultation confirms positive views on performance
ESF was most effective in supporting PES and LM institutions; quality of higher education and social inclusion systems
Institutional capacity building: success was highly dependent on political support, mutual learning, netwoking and exchange of experience between stakeholders
Promotion of Partnerships: success depended on integration to national strategies, proper needs asssessment, and MS capacity to develop concepts and services were key to success
The average cost per participant is about 900€, however there is a wide dispartity of costs. This depends on factors such as the policy theme (higher in A2E and SI), role of the ESF (higher for innovative interventions or support to full national implementation of minstream measures),
The evaluations did not cover administrative burden issues (ESF was coverd by DG REGIO evaluation on Delivery Systems), but the public consultation did. Result point to management and control systems, reporting and audit as the more burdensome areas,
Audit requirements credited for reduction in error rates
"single audit" principle not always respected
Simplified Cost Options associated with eased burden but limited use in

15. Weak evidence on sustainability of results, varies considerably
Scarcity of follow-up data on participants measuring sustainability of effects
Where it exists, there is considerable variety for individuals, ranging from % depending on the nature of intervention and target group
Mixed results were found for actions on systems, with the exception of investment in LLL systems
Sustainability of systems dependent on conversion of new working methods in lasting networks, sharing of lessons learned and mainstreaming of approaches

16. ESF EU value added
Volume: additional resources to support EU and national employment policies
Help weather the effects of the crisis
Scope: Extension of support to new target groups and offering of more tailored services
Disabled; Young; Unemployed with low qualifications
Role: test and implement innovative activities and reforms to employment and social services, education and training systems
Process: support administrative capacity building
PES (Public Employment Services)
Educational institutions and Healthcare
Apart from reaching a signficant number of EU citizens, there are other areas in which ESF has demonstrated EU value added:
The ESF has provided additional resources to support national policies. In number of countries ESF finaced majority of ALMP activities (EL, SK, LT). In other ESF covered large proportion of pupils and students at all levels of education (BG, PT),
The ESF prevented more negative effects of the crisis and helped MS weather lack of funding during the crisis and maintain investment in education, ALMP and social inclusion
The ESF has enabled the extension of support to new target groups and offering more tailored and intensive services to specific groups
The ESF has supported innovation and the introduction of reforms in education and training (i.e.: new curricula, apprenticeships, VET reforms), public employment services, and reforms in the public administration (i.e.: reforms in Justice, e-governance and enhancing buisness friendlines of the administration).
As a conclusion, the ESF has been an effective instrument to develop a human face of Europe, providing tangible benefits to people and contributing to a more competitive and inclusive Europe,

17. Lessons learned Policy choices: Programming: Target groups:
Continue aligning ESF with EU/national priorities
Flexibility to adjust to emerging needs
Programming:
Robust definition of objectives, targets and results
Apply more evidence-based programing
Target groups:
Ensure coverage of disadvantaged groups
Continue focus on young and old and balanced representation by gender
Programme Implementation:
Promote customisation to the needs of specific target groups
Improve capacity building
Further simplify procedures and continue reducing administrative burden
Monitoring systems:
Higher standardisation of programme indicators
Improve use of longitudinal and micro-data
Capture effects other than employment and qualifications – "soft results"
Evaluation:
More robust impact evaluations
Reintroduce final evaluations/timing of evaluations
Continue aligning ESF with EU/national priorities and link with objectives as expressed by CSRs needs to maintained in the forthcoming programming periods
Keep flexibilty so that programmes can be adjusted on the face of unexpected emerging needs
Deepen the results orientation by strenghtening the definition of objectives, targets and results and applying more evidence-based programming
Specific actions need to be taken as to ensure coverage of disadvantaged groups (Roma, migrants).
Continue focus on young and old people and balanced representation by gender
Improve capacity building at all levels
Further simplify and continue reducing administrative burden:.
Improve use of longitudinal and micro-data - Capture effects other than employment and qualifications
Higher standardisation of programme indicators
Review evaluation provisions to foster impact evaluations and reintroduce final evaluations

18. Changes in the 2014-20 regulations
Alignment of ESF with EU priorities – CSRs: thematic objectives in line with Europe thematic ex ante conditionalities
Enhanced flexibility to adjust programmes
Coordination with other ESIF: Partnership agreements + multifund Ops
Programmes have to set specific objectives translated into clear indicators of results with targets and benchmarks
Regular reporting of results (broken-down by gender) and outputs and a performance framework linked to a performance reserve
• for projects up to 50,000 euros

19. Changes in the 2014-20 regulations
Use of SCOs expanded (increased legal certainty and mandatory uptake)
YEI integrated into ESF
20% ESF in SI – coverage of disadvantaged
Impact evaluation for each of the specific objectives
Timing of evaluations: summary report of MS evaluations by December 2022

20. Lessons requiring attention from COM on 2014-2020 and post 2020
Encourage MS to report on "soft results"
Increase uptake of simplification
Continue building MS evaluation capacity
Better capture the results of capacity building activities of the programme
Support and promote use of FEIs
Streamlining application of the single audit principle
Consider the recommendations of the High Level Group on simplification
Other lessons require attention during the implementation period and post-2020:
 ‘Soft’ results were not considered in the common monitoring framework for , but the Commission should encourage Member States to use them, in particular for the purpose of impact evaluations;
 The Commission supports Member States’ evaluation capacity on an ongoing basis through its regular meetings with evaluations partners and the ‘community of practice’ by CRIE as well as evaluation helpdesk. In these forums, the Commission fosters the combined use of micro-data and administrative registers to carry out longitudinal studies. It will also seek to address issues such as capturing the effects other than employment and qualifications fostering the use of macro-economic models to estimate impacts;
 The Commission will also explore possibilities to better capture the results of capacity building activities, for example by fostering the measurement of empowerment;
 The Commission will continue its efforts to persuade Member States to increase the uptake of simplification procedures and reduce administrative burden for beneficiaries, MAs and implementing bodies, such as the recently proposed Delegated Act on SCOs and the simplification of reporting on households in the framework of the ‘Omnibus Regulation’81;
 FEI implementation was disappointing and hampered by a slow disbursement of funds and by reporting arrangement deficiencies. Some issues were addressed in (e.g. through more detailed reporting arrangements) and periods, but there is a continued need for support and promotion in the use of FEIs for future programming periods;
 Although progressive simplification measures were introduced in , the overall level of uptake remained low. The mandatory use of SCOs for small projects in should be seen as a step in the right direction. Further ways in which the authorities can improve legal certainty should be sought, so as to address the concerns on the part of MAs that prevent the widespread adoption of SCOs;
 Although the strengthened audit framework resulted in a reduction of reported error rates as compared with previous programming periods, audit remained one of the areas identified in the open public consultation and the DG REGIO evaluation on delivery systems as excessively burdensome. Means of streamlining the application of the single-audit principle should be considered in forthcoming programming periods; and
 The Commission will consider the recommendations of the high-level group of independent experts on monitoring simplification for beneficiaries of the European structural and investment funds.
This evaluation will feed into discussions on post-2020 and any impact assessment for future programming periods.

21. Thank you! For more information on ESF: