ESF 2007 – 2013 Ex-post evaluation: Supporting the integration of disadvantaged groups into the labour market and society Prepared for: ESF Evaluation Partnership 10 June 2015 Richard Lloyd, ICF Richard.Lloyd@icfi.com

Overview Context for the 2007-2013 programme period Key findings – 27 Member State overview: Structure/scale of investment Outputs and results Analysis of eight in-depth Member States: Effectiveness and what works Efficiency and sustainability Gender sensitivity Community Added Value Key lessons

Context for the 2007-2013 programme period Overview of 27 Member states Context for the 2007-2013 programme period OPs developed after a period of economic expansion, followed by the crisis: Particular impact on those facing disadvantage…. Rate of LTU and VLTU At risk of poverty rates Crisis led to adjustments in ESF programming/re-direction of resources: As part of wider Member State recovery packages Primarily towards disadvantaged groups including the long-term unemployed facing labour market disadvantage, young people NEET/at risk and disabled individuals

Structure and scale of ESF investment in SI Overview of 27 member states Structure and scale of ESF investment in SI SI investment positioned differently in and within OPs and Priority Axes: Most commonly a single PA, or sub-PA, dedicated to SI; other PAs focused on SI and A2E; in others SI ‘mainstreamed’ across PAs – but vulnerable groups targets across PAs Structural diversity a challenge - importantly for monitoring ESF SI investment SI investment reflected the Member States priorities – most commonly: Reforms/measures to enhance PES capacity (18), reinforce ALMPs (11), and reform social benefits/income support measures (10) Reforms to social services (3) and fiscal disincentives for low or second earners (4); Develop strategies/policies to support youth (4), social economy (1) and public works (2) Level of investment in SI to end 2013, shared SI and A2E PAs (SI PAs alone): Total certified ESF expenditure is €31.5 billion (€7.4 billion for ‘pure’ SI PAs); Represents 45% (or 11%) of ESF expenditure – in both cases implementation rates >60% Comparing ESF SI to national SI expenditure challenging: ESF SI and A2E PAs account for 2.7% of labour market spend (0.45% for ‘pure’ SI PAs) – and over 10% of national labour market spend in seven MS

Outputs - participation in ESF SI actions to end 2013 Overview of 27 Member States Outputs - participation in ESF SI actions to end 2013 Wide range of target groups – aligned with Member State priorities - commonly: Over 21 million ‘participations’ under SI and A2E PAs (6 million SI alone): Of the SI and SI/A2E participations - 64% were unemployed (19% LTU) and 23% inactive 52% were female, 27% young people, 7% aged 55-64, and 51% low levels of education But fewer than expected classified as ‘vulnerable’ groups – Annex XXIII data shows 12% as migrants, 8% as minorities, 11% as disabled and 13% ‘other’ (just 44%)

Results - to end 2013 Achievements of SI focussed actions: Overview of 27 member states Results - to end 2013 Achievements of SI focussed actions: Over 450 result indicators – many without targets (171), some expressed as numbers and others as percentages Performance against results targets to end 2013 variable – where set majority ‘achieved/on target’ Headline achievements (SI, where indicators/data available): Based on aggregate numbers and average of percentages as available Few examples of soft result data being routinely collected and consolidated: Particularly relevant for SI – collected by around 25% of the in-depth interventions Results Numbers Member States Percentages Accessing employment 1.58 million 9 33.2% 7 Achieving qualifications 342,576 5 64.5% 3 Other positive results 1.30 million 11 44.7% 8 Total 3.2 million --

Key findings – effectiveness Analysis of in-depth countries Key findings – effectiveness Across the 58 interventions reviewed, many still ongoing: Financial implementation rates ranged from 83.5% to 91% - holistic actions highest Over 250,000 participations reported, of which: 29% achieved an immediate employment outcome (highest in clusters 2a and 2b) 26% achieved a qualification (2a), and 40% another positive result (2a and 2b) However achievement underestimated as soft result data rarely collected/consolidated And surprisingly low representation of ‘disadvantaged groups’ (14%) Proportion of interventions meeting/exceeding output and result targets

Key findings – what works/good practice Analysis of in-depth countries Key findings – what works/good practice No clear pattern of what works (and for whom) – but a series of principles and ‘good practice’ points identified For interventions these include: Holistic approaches around a pathway of interventions supports the integration of the most disadvantaged groups Activities planned around this pathway - with support to engage, robust assessment and action planning procedures Actions/interventions to overcome barriers to participation, with a clear focus on next steps towards employment and with follow up support Tailored guidance and training can take account of existing skills and competences while addressing personal, social or skills barriers Support for employment creation can be particularly helpful for groups faced with prejudice or which productivity may (initially) be restricted Follow up support post placement is critical - but rarely delivered as a single measure Systemic support ties in with the need to modernise service delivery and improve partnership working

Key findings – efficiency and sustainability Analysis of in-depth countries Key findings – efficiency and sustainability Efficiency – within constraints of data availability and ongoing delivery: Average cost per output varied significantly –between €3,343 (cluster 4/systemic) and €5,519 (cluster 2b/employment as an outcome) Average cost per result also varied between clusters and results: Employment results from €16,076 (cluster 2b) to €105,310 (cluster 1/supporting-enabling) Qualification results from €7,500 (cluster 2b) to €121,139 (cluster 1) ‘Other positive outcome’ results from €6,002 (cluster 2a/IAG, training) to €22,625 (cluster 1) Sustainability – in terms of results for individuals and of interventions: For individuals: Few routinely measure outcomes after 6 months, follow-up limited and lack of interaction with administrative databases Where measured sustainability varies – 19% to > 70% for employment results (n=8) For activities: Actual/expected sustainability rate high – over 80% Frequently linked to new ESF programme – 67% of those actual/expected to continue Examples of ESF interventions, wholly or in part, transferring to mainstream funding or influencing practice

Key findings - gender sensitivity Analysis of in-depth countries Key findings - gender sensitivity Context: Background data shows that amongst the wider SI target groups, women are often affected by multiple disadvantage Almost equal representation of women and men in the in-depth interventions: Female participation 46% for in-depth interventions (vs. 52% in all SI interventions) Some variation by cluster – most pronounced for holistic interventions (cluster 3) However, little evidence of gender impact assessment in OPs, intervention selection or planning: Despite emphasis on gender sensitivity and gender mainstreaming in the Regulation

Key findings - Community Added Value Analysis of in-depth countries Key findings - Community Added Value Volume effects most clearly identifiable, although others evident Volume effects: Although ESF share of national SI expenditure not clear, many individuals received support who would not have done so otherwise Scope effects: Provided tailored support for target groups not otherwise able to access services (e.g. female migrants; female sex workers; young care leavers; substance users) Role effects: Local models mainstreamed into provision with national or additional ESF funding (e.g. key workers; young people transitions; peer mentoring) Process effects: Establishment of local partnerships, between public and private sectors, to support SI target groups – seen as a key success factor for several interventions

Key lessons included…. Policy choices: Analysis of in-depth countries Key lessons included…. Policy choices: Personalised/tailored interventions key – ESF helps ensure targeted to those in most need ESF can help modernise labour market institutions/practices to better meet needs Target group selection: Broad target groups in OP can help adapt to change Categorisation issues – consistent data required Finer grain of evaluation studies required to identify what works for which groups Programming: ‘Mainstreaming’ enables flexibility – monitoring Linkage with national/EU policy priorities allows gaps to be filled and opportunities exploited Gender sensitivity an area of continued learning Plan from outset with a focus on employability Implementation: Quality of staff key Intensive support, throughout intervention, key Focus on staff/client relationships (trust) Referral/outreach and engagement approaches key Celebrate achievement Monitoring and evaluation: Shortcomings in MI data: definitions, target setting, link to admin(for CEI), Annex XXIII and AIR accuracy Specific issues for SI actions – isolating interventions, agreeing core indicators, soft results, follow-up Evaluation of SI limited and CIEs very rare – must be prepared for in advance Throughout – importance of partnerships key in mobilising support/engaging with disadvantaged groups, engaging partners and stakeholders from the outset and throughout delivery

Aims and evaluation approach To assess the achievements of ESF investment in supporting the integration of disadvantaged groups into the labour market and society – including: The distribution and effectiveness/efficiency of ESF use; The socio-economic impacts resulting and their sustainability; and Good practice/key success factors – to inform future investment. Two main components: Review of the SI investments across 27 Member States – strategic priorities, mapping Priority Axes/activities/target groups, expenditure, outputs and results at PA level; and Detailed review in eight Member States – strategy and OP architecture, review of 58 interventions (effectiveness, efficiency, sustainability, gender sensitivity, CAV). Data sources/methods: 27 Member State review – mapping PAs, activities/target groups; analysis of SFC and 2013 AIR; In-depth assessment – monitoring/AIR data, interviews with MA, intermediaries, providers and stakeholders. Established four ‘clusters’ of activity – supporting/enabling, employment as outcome, holistic/pathways and systems measures

Interventions and activities – four clusters Four main and three sub-clusters of ESF SI activity were established: