ESF Expert Evaluation Network 2007-2013 Evaluation Partnership Meeting 18.11.11 Herta Tödtling-Schönhofer & Isabel Naylon

Objective To set up an Expert Evaluation Network with one national expert in each Member State which collects, describes and analyses all evaluations related to ESF Operational Programmes carried out by the Member States in the 2007-2013 programming period Main task To produce Inventories of evaluations and Country Reports for all 27 Member States and a Synthesis Report at EU level summarising the evaluation findings

1. Evaluations

Evaluations By July 2011, national experts had identified 559 evaluations, whereby distribution is very uneven across the Union Ongoing evaluations have been included in the inventories but not in the analysis Only finalised evaluations (final reports) were screened for findings except in the case where interim evaluations had the character of final evaluations (were on specific measures, would not be updated etc.) Most evaluations are carried out at OP level

Evaluation methodologies In keeping with the aim of the evaluations (operational), 55% were exclusively process evaluations, 18% exclusively impact, 10% were both impact and process, 17% were other 122 were impact evaluations (of which 44 were both impact and process); of these 45 were theory based 26 used a counterfactual approach 24 were classified as ‘other’ and 27 as unclear 10 MS have used the counterfactual approach so far Most evaluations carried out at OP level

Key figures 112 Evaluation Plans 559 Evaluations … of which 117 Planned 84 Ongoing  330 Finalised  28 Unclear 233 Process evaluations (exclusively) 78 Impact evaluations (exclusively) 139 Strategic evaluations 307 Operational evaluations Total findings 1651 Methodology Approach

Evaluations by Member State

2. Evaluation Findings sorted by…..

Findings by evaluation criteria

Findings by intervention type

Findings by target group

Findings by individual target groups

Findings by policy field

3. Evaluation Findings are telling us….

a) Increased adaptability /1 Findings focus on the: establishment of new enterprises growth of SMEs organizational learning, global management quality development of innovation The findings concern employed people (40%) self-employed people (22%), older people (10%), unemployed (8%), other disadvantaged groups (9%) and to a lesser extent inactive people, young people, women and people with disabilities

Increased adaptability /2 Support is provided through: training activities (65% of all findings) start-up incentives (10%) and education (9%) and to a lesser extent through LM services, employment incentives and social inclusion specific support, etc. Evaluations provide a positive assessment of the relevance and effectiveness of ESF interventions in this field, in particular in the context of the financial crisis The ESF fosters the development of innovative and more productive forms of work

b) Enhancing access to employment /1 Findings mostly concerned with unemployed people (31%) and young people (20%), to a much lesser extent self-employed, inactive and older people, people with disabilities, Few evaluations to date focus on migrants and Roma The financial crisis has increased the relevance of ESF interventions in this field but has also hampered their effectiveness, in particular for the most disadvantaged target groups

Enhancing access to employment /2 The evidence concerning the modernisation and strengthening of labour market institutions is rather mixed The most effective interventions for the unemployed are close to the labour market and have employer involvement Proximity to the users is a success factor Barriers to participation remain, in particular for target groups most distanced from the labour market Actions fostering gender mainstreaming seem to be successful in raising women’s participation rates

c) Reinforcing social inclusion Some target groups are not reached by the programme despite targeting by the ESF due to ‘creaming off’ strategies Evaluations so far show that success factors in the field of social integration include: integrated strategies across policy sectors (health, social affairs, education) Local interventions and partnerships at local level Personalised services The effectiveness of social inclusion measures ranges from economic returns to the individual well-being of participants

d) Improving human capital The ESF interventions contribute to reforms in education and training systems The findings indicate good progress in lifelong learning and training, in particular in the introduction of new products in enterprises In particular in the light of the financial crisis are measures fostering the development of human capital found to be relevant

e) Partnerships and networking The evaluations so far hardly deal with the promotion of partnerships and networking (only 2% of the findings address this policy field) Two main strands emerge when analysing the results attributed to partnerships across Member States: Partnerships have an important function in increasing awareness of the local dimension Partnerships and networking allow diversification across fields and actors involved A number of challenges are identified in the capacity of Member States to develop partnerships, e.g.: concerning the involvement of the private sector and the development of clear goals

f) Strengthening institutional capacity According to the Regulation 1081/2006, the ESF supports actions to strengthen institutional capacity by promoting mechanisms for improving good policy and programme design, monitoring and evaluation at national, regional and local level, capacity building in the delivery of policies and programmes Evaluations looked at both the administrative capacity actions and at weaknesses in administrative capacity in the implementation of Operational Programmes This lead to some confusion as to which findings strictly belong to the policy field and which do not but have been put here for want of a better category

Strengthening institutional capacity /2 Where it was clear that the findings were not linked to the Policy Field ‘Strengthening institutional capacity’, they were allocated to ‘All Policy Fields’ – again for want of a better solution Findings that can be linked to this policy field stem to a large extent from evaluations implemented in Poland (121 evaluations and 235 findings attributed to this policy field) due to the administrative reforms introduced in Poland

4. Synthesis and ConcLusions….

Conclusions /1 The manner in which Member States have used the flexibility of the new regulations and the evaluation approach they have adopted vary considerably Evaluations in some Member States, especially in the EU12, tend to focus on implementation and operational aspects The sum total of the evaluations at the end of the period may not give a coherent overall picture of the effects and effectiveness of the ESF across the EU Evaluations in the current period may tend to focus more on the deficiencies of programmes co-funded by the ESF than on the benefits of the ESF as such

Conclusions /2 The ESF is a useful financial instrument, especially for those Member States in which the ESF represents a significant proportion of funding for LMP and in the current context of economic downturn The ESF contributes to the adaptability of workers to the labour market mainly through training activities The ESF contributes to enhance access to employment for the unemployed and young people but disadvantaged groups still face barriers to participation

Conclusions /3 The ESF contributes to social inclusion but effects on the most disadvantaged groups are not yet evaluated in all the Member States The ESF contributes to public administration reforms in Convergence Member States and regions The contribution of partnerships and networking to the improvement of reforms and labour market policies in the MS is little evaluated The ESF supports the MS in addressing horizontal priorities (gender mainstreaming, sustainable development, and transnationality) but these have hardly been evaluated so far

Conclusions /4 CAV and sustainability of ESF interventions have not yet been evaluated There are still weaknesses in the ESF monitoring and evaluation: link between monitoring and evaluation data quality data collection methods information about beneficiaries indicator systems with target levels link between projects and OP

Thank You Benno Savioli, Feedback Bremen Herta Toedtling Schoenhofer & Isabel Naylon, Metis GmbH schoenhofer@metis-vienna.eu naylon@metis-vienna.eu Hélène Génin & Irene Basile, Euréval genin@eureval.fr basile@eureval.fr Benno Savioli, Feedback Bremen savioli@feedback-bremen.de