1. ESF Ex-Post Evaluation 2007-2013 TWG Amsterdam 13 June 2016

2. The ESF in
Evaluation questions and methods
Key preliminary findings
Lessons learned and next steps

3. The ESF in

4. ESF objectives
Strengthen social and economic cohesion: employment opportunities, more and better jobs
Supporting MS Policies on the basis of the European Employment Strategy guidelines and Country Specific Recommendations
Please note that in the next slides, Human Capital (or HC) will be understood as regrouping enhancing Human capital and adaptability of workers
Increasing Adaptability of workers (HC)
Enhancing Human Capital (HC)
Social Inclusion of disadvantaged(SI)
Enhancing Access to Employment (A2E)
Institutional Capacity (SIC)
Promoting Partnersh (PP)

5. ESF programmes in
115.6 billion € allocated, of which 77 billion € EU funding
7,9 % of total EU budget - 8,5% in
All MS, Convergence & Regional Competitiveness and Employment (RCE) regions
Shared management: MS program & manage expenditure
117 Programmes, each with several priority axes
Concentration through implemention of CSR

6. Evaluation questions
and methods

7. Evaluation questions Art. 49.3 of Council Regulation 1083/2006
Extent to which the resources were used
Effectiveness
Efficiency
Socio-economic impact
Additionnal criteria:
Community Added Value
Gender sensitivity
Sustainability
Lessons Learned
Above, we mention the criteria required by the regulation, but the DG decided to add some criteria to complete the picture. All these criteria taken together will respond to the requirements of better regulation (which were adopted when the evaluation was already ongoing)

8. Evaluation deliverables
24 Clusters of interventions
Thematic studies
Synthesis study
Preparatory study
Human capital
Update 2014 data +
Country reports
SIC and PP
Croatia
Commission
Staff
Working
Document
Access to employment
NB: evaluation not finalised yet, some quantified figures subject ot chang, especially results!
Preparatory study carried out in 2013 on how to best meet the regulatory requirements, leading to 3 main themes (covering 4 ESF priorities)
: Investment in Human capital (also includes adaptability of workers), Supporting the integration of disadvantaged groups into labour market and society, access and sustainable integration into employment Followed by a Synthesis updating the performance data, with country reports, including Croatia and support to partnerships and institutional capacity building
Finally, the results of the ex-post will be presented by the Commission in a Staff Working Document, which will include also results from the Public Consultation
Evaluation method : for each of the three thematic studies
Mobilisation of all existing evidence both from Annual Implementation Reports and from evaluations from MS
Complemented by an in-depth analysis of a selection of interventions in some MS - each MS covered by at least one thematic evaluation
For analytical purposes, selected interventions were grouped in clusters which were meant to reflect the variety of intervention approaches/types
Open Public Consultation
Social inclusion

9. Limitations with data and evaluations
Priority Axis in the OP not matching priorities of the ESF Regulation
Data problems despite improvements related to Annex XXIII:
Data on participations, not participants
Sometimes indirect participants reported
Disadvantaged groups underreported
Incomplete reporting on socio-economic characteristics
Data reported only at priority level
Lack of common definitions and common results indicators
Evaluations from MS providing little evidence on impacts
Limitations faced in this evaluation and how these were addressed
-design of Programmes: MS were free to formulate the priority axes (Pax) under the programme, the evaluators therefore allocated each Pax to a single policy theme for the purpose of the evaluation, although some tackled several themes
- data on participants: although the evalaution could draw for the first time on common indicators related to participations (so called annex xxiii) to aggregate at EU level, the participants data also included indirect participants (to a systems support), and double counting if interventions were split, pointing to an overall overreporting. Disadvantaged groups on the other hand were under-reported, notably due to legal restrictions. Evaluators were therefore prudent in their conclusions.
data relating to results: for participants, it was possible to aggregate a fair share of results in three categories:gaining or maintaining employment, obtaining a qualification or other positive results (skills, competences…). Also calculation of estimated success rates excluding priorities where no result could be aggregated.
limitations on evaluations from MS: although the Commission managed to create a database on all evaluations carried out by MS (over 700), these were mainly process related and included very few impact evaluations and even less quantitative ones (so called counterfactual impact evaluations). Those available were used.

10. Weaknesses in monitoring systems hinder a more robust assessment
Not all results could be aggregated (73% of budget)
35% of results indicators didn't have a target and 8% were not monitored
In most cases lack of results and targets by gender, and for youth
Soft outcomes, crucial for SI, were rarely measured
Scarce data/evaluations on actions supporting systems
Good practices difficult to extract due to lack of data at intervention level
Even if a majority of the results indicators have reached their targets, 35% of the result indicators did not have a target in the OPs, and another 8% were not monitored
In most cases, gendered results and targets were absent
Similarly, results for youth specifically were rare
Soft outcomes, crucial for SI, were rarely measured
Evidence was also scarce for the interventions analysed in depth, especially on follow-up, thus good practices were difficult to extract
√ Findings are mostly qualitative

11. Key preliminary findings

12. Strong alignment of ESF to EU and national challenges and priorities
ESF was found highly relevant for responding to EU headline targets and EU policy guidelines
Challenges identified by CSRs well reflected in Programmes
Flexibility in programming allowed a swift response to the crisis and without a need to formally change programmes
Public consultation: most agreed that the ESF responded to key challenges for HC, SI, A2E, while SIC and PP were less recognised
ESF supported the implementation of national reforms in MS, which responded to EU priorities first under Lisbon, then EU 2020, as well as to the SI recommendation and the Education and Training 2020 strategy
Even overall available funds were reduced as a result of austerity measures taken by some MS

13. Most of ESF allocated to Convergence regions …..
68% of EU funding to Convergence regions (58% of total ESF)
More on investment in HC systems
More on mainstream interventions
31% of EU funding to RCE regions (42% of total ESF)
More in access to employment and social inclusion
More on new/innovative interventions
Significance and role of ESF varies substantially:
Across MS: higher in new MS
Policy fields: higher for A2E and Social inclusion
ESF in CONV: 68% of the EU contribution // 58% of the total ESF value of the programmes (EU+national)
ESF in RCE: 31% of the EU contribution // 42% of the total ESF value of the programmes (EU+national)

14. … and investment in human capital and adaptability
Please note that this refers to budget allocation, it was based on the EU funding.
Public consultation: most agreed that the ESF responded to key challenges for HC, SI, A2E, while SIC and PP were less recognised

15. Implementation has progressed adequately
91,7 billion spent by end of 2014 (79,3%) billion
No significant difference in implementation rate in different type of eligible regions or policy fields, however
implementation in PP and SIC significantly slower,
and in some countries (HR, RO, SK, MT)
By end 2014, 63% of the result indicators had exceeded their targets or were about to reach them
Factors explaining low absorption mainly related to administrative capacity issues:
Lack of effective coordination between IB and project implementers
Longer than anticipated project preparation
Problems with obtaining national co-financing
Lack of management capacity by project promoters
Difficulties to reaching target groups
54% of the result indicators hade reached higher results than set in their targets
9% were in the % achievement rate
These figures will certainly increase by end of the period,
NB: this is based on total figures, that is EU funding and national co-financing

16. ESF has reached most relevant target groups of participants
98 million participations till end of 2014
Convergence 63%, RCE 37%
Low skilled (45%) and inactive (36%), people in employment (33%), youth (30,5%), and unemployed (30%)
Decreasing relative share of unemployed, migrants, minorities
Relative share of youth and older workers remained constant

17. ESF participants by policy themes

18. in achieving results for participants
ESF was effective
in achieving results for participants
At least 30.5 million positive results …….
8.9 million into employment
8.6 million gained a qualification
12.8 mlliion obtained other positive results
0.3 million self-employed
……… related to 74.5 million participants, overall results ratio of 41%
Public consultation: positive views on the results achieved by individuals with the support of the ESF
Please see limitations on results
Success rates from the in-depth analysis of interventions are higher (55 % for example for HC)
Respondants in the open public consultation (285 replies received, 256 could be analysed) were positive on the results achieved in gaining qualifications, employment, improving skills of teachers and educators, and working conditions of the employed

19. Support to individuals was generally effective in all policy themes
Human Capital:
early school leaving and transition to LM
participation in higher education
upskilling of adults
Access to Employment:
personalised support
training
migrant participation
Social Inclusion:
pathways and direct employment
The se are the most effective ones per policy area, as assessed by the thematic evaluations

20. ESF also achieved results
in supporting systems
Over 200,000 entities obtained a positive result
Over 100,000 products /systems/tools developed
Public consultation: positive views on ESF supporting the development of new qualifications, courses, training programmes, standards or systems, rising the competitiveness and adaptability of enterprises, start-ups and improving public administration effectiveness and efficiency

21. Though it was less successful than supporting individuals
Most effective in supporting PES and LM institutions; quality of higher education and social inclusion systems
Institutional capacity building: success was highly dependent on political support, mutual learning, netwoking and exchange of experience between stakeholders
Promotion of Partnerships: success depended on integration to national strategies, proper needs asssessment, and MS capacity to develop concepts and services were key to success
Even less data and evaluations were available for systems than for individuals (at OP and intervention level)
Overall clusters relating to system support analysed within HC, SI, and A2E are less successful than those targetting individuals, as they take longer to bear fruits

22. ESF interventions have been generally effective
There are clear limitations affecting the quality of the evidence available :
data quality concerns
Most national evaluations were process oriented
But, bringing all available evidence toghether (triangulation), ESF can be considered effective
Even if a majority of the results indicators have reached their targets, 35% of the result indicators did not have a target in the OPs, and another 8% were not monitored
In most cases, gendered results and targets were absent
Similarly, results for youth specifically were rare
Soft outcomes, crucial for SI, were rarely measured
Evidence was also scarce for the interventions analysed in depth, especially on follow-up, thus good practices were difficult to extract
√ Findings are mostly qualitative

23. Cost per ESF participant widely variable
Average cost per participant 900 € par year but wide variations exist:
By policy theme: lower on HC, higher for A2E and SI
Depending on role of ESF: lower if ESF complements national policies or higher if used to develop innovative approaches
Systems support and groups with specific needs more expensive
Public consultation: positive on cost-effectiveness
National ALMP cost per person in the same period: 5,600 € per year, but caution before saying ESF is much cheaper as this should rather be assessed country by country

24. How expenditure compares to participants
The average cost depends on:

25. Admin burden high in some areas
Public consultation:
positive as regards communication, implementation of projects, designation of authorities, design of OP, selection of projects, and evaluation
areas deemed excessively burdensome: management and control systems, reporting, and audit
Confirms partially results from the evaluation of delivery systems (conducted by DG REGIO)
single audit credited for reduction in error rates
Simplified Cost Options associated with eased burden
Financial instruments used in 7 MS (0,6% of total EU funding)
This was confirmed in the evaluation of delivery systems conducted by REGIO, but for project selection and implementation (rated as high due mostly to gold plating)
At the same time focus on audit credited for reduction in error rates (from 10% in to 3% for ESF currently).

26. Weak evidence on sustainability of results, which vary considerably
Scarcity of follow-up data on participants measuring sustainability of effects
Where it exists, there is considerable variety for individuals, ranging from % depending on the nature of intervention and target group
Mixed results were found for actions on systems, with the exception of investment in LLL systems and training of staff which proved sustainable at national level
Sustainability of systems dependent on conversion of new working methods in lasting networks, sharing of lessons learned and mainstreaming of approaches

27. Gender sensitivity
Relatively balanced participation by gender (51,4 % women)
Results were not declined by gender
Most applied the principle of gender equality as a horizontal principle
Most interventions did not include specific actions addressing gender sensitivity
Lack of evidence to assess articulation of gender equality in intervention planning, design and delivery

28. Youth 30.1 million participations of young people (30.5%)
Estimated investment of 35.4 billion Euro
Despite emphasis on young, few specific indicators, especially on results
Wide diversity of approaches: specific or broader
Most specific interventions focused on improving employment opportunities and long-term position in the labour market of low skilled young people

29. Few examples of coordination with other EU Funds
Few examples of inter-programme coordination between ESF and ERDF, in some MS none
Prominent exceptions: entrepreneurship and Human Capital, and sometimes in the areas of social inclusion, institutional capacity and promotion of partnerships
Public consultation: many did not respond to the question of complementarity, indicating that few had experience with other EU funds

30. Considerable EU value added through volume and scope effects ……
Provision of additional resources to support national policies (volume effects) is the most visible source of value added in all policy fields:
Most visible in new MS, convergence regions
Even in MS with low ESF significance volume effects achieved in specific domains (e.g. early school leaving, school education, transition to the LM).
Particularly important in the context of the crisis
Significant scope effects also in all policy fields:
Broadening the support to new target groups
Offering more tailored and intensive services to specific groups

31. …. but also through support to new approaches and administrative reforms
Role effects more visible in the field of education and training:
New methods of teachers' training and curricula
But ESF has been used to test new and innovative activities in all policy fields
Process effects:
Mainly found in PP and SIC to build the administrative capacity services (mainly in convergence regions),
Structural reforms in PES and educational institutions
Partnerships for social inclusion
Public consultation confirmed that ESF was instrumental in testing and implementing innovative activities in all policy themes

32. Socio-economic impacts mostly at micro meso-level
Largest effects observed at micro level:
large number of participants and results,
in convergence regions, ESF reached between 3 to 21% of the students in the various ISCED levels
Meso level effects reflected in process and role effects
Increased scope of mainstream social services
Implementation of innovative approaches
Capacity building and modernisation of public services
Significant correlations were found with the national share of people at risk of poverty suggesting MS with higher significance of ESF made more progress in reducing the number of people at risk

33. more difficult to establish
Impacts at macro level
more difficult to establish
Insufficient evidence of ESF link to wider macro- economic impacts
Considerable improvements in areas targeted by ESF: ESL rates, higher education attainment, gender gaps in education indicators
Significant correlations were found with the national share of people at risk of poverty
ESF helped compensate lack of national funding due to budgetary constraints
Public consultation: most agreed that the ESF helped mitigate the economic and financial crisis

34. Lessons learned

35. Lessons learned Policy choices: Programming:
Continue aligning ESF with EU/national priorities
Find balance between support to innovative and mainstream activities
Flexibility to adjust programmes to emerging needs
Programming:
More robust definition of objectives supported by measurable targets for outputs & results
Target setting: develop a common methodology
Clearer demarcation of policy fields and types of intervention
Apply more evidence-based programing
Assure gender sensitivity and sustainability

36. Lessons learned Target groups: Programme Implementation
Improve outreach to disadvantaged groups
Target specific groups least covered by national measures
Promote customisation of interventions to the needs of specific target groups
Programme Implementation
Improve capacity building of project promoters
Involve social partners in design and implementation
Clear instructions and guidance on reporting
Improve MIS and information on successful interventions

37. Lessons learned Monitoring systems Evaluation
Higher standardisation of programme indicators
Collect participants data over time using administrative data
Improve individual project data
Better capture soft outcomes
Evaluation
More robust impact evaluations
Use broader evaluation frameworks to capture the benefits of supporting disadvanted groups
Reintroduce the concept of final evaluation
Review timing of the ex-post evaluation

38. What next? Finalisation of last packages
Preparation of SWD (September)
Disseminate results, starting with Cohesion evaluation conference in Sofia (16-17 June)
Use evidence to support the impact assessment of post 2020 ESF