Guidance document on ex-ante evaluation 15 March 2012 Joint presentation to the ERDF and ESF Evaluation Partnership Meeting

Ex ante evaluation: our approach What is different? Common Regulation & multi-fund programmes Stronger focus on results / Europe 2020 strategy More detailed Article 48 Links with Partnership Contracts, SEA Our common approach: Stick to Regulation (article 48 and related articles)-as proposed by Commission Focus on ex ante evaluators' role When necessary, give examples of differences between ESF/ERDF

Ex ante evaluation Guidance Programme strategy Q&A Indicators, monitoring and evaluation Consistency of financial allocation Contribution to Europe 2020 Strategic Environmental Assessment Evaluation Process

Programme strategy Consistency of programme objectives Coherence Horizontal principles Linkage between supported actions, expected outputs and results Q&A

Programme strategy – key reference points Art. 24(1) CPR: Europe 2020 strategy Common Strategic Framework Partnership Contract Art. 87(2)(a)(i) CPR: Country specific recommendations national and regional needs

Consistency of programme objectives Step 1: The ex ante evaluator should assess the challenges and needs identified by the programme. Step 2: The evaluator should examine if these challenges and needs have been consistently translated into thematic objectives, priorities, investment priorities and corresponding specific objectives.

Step 1: Challenges & needs (1) National & regional programmes: the evaluators should primarily base their assessment on the NRP, country-specific recommendations, European semester analyses In addition: National ERDF sectoral programmes: specific sectoral analysis & territorial priorities Regional programmes: specific regional situation and needs Specific challenges of sub-regional or functional areas or of specific target groups Horizontal principles (Art. 7 & 8 CPR) Justification of specific regional challenges and needs

Step 1: Challenges & needs (2) The ex ante evaluator should assess whether the identified challenges and needs are consistent with the analysis of disparities and development needs in the Partnership Contract. The evaluators should also base their assessment on the Common Strategic Framework. They should examine the prioritisation of the identified challenges and needs and suggest reviewing it, if appropriate.

Step 2: Programme objectives thematic objectives investment priorities Specific objectives While appraising the consistent translation of the identified challenge & needs into programme objectives, the evaluators should focus on: Their appropriate weight in the investment priorities Precise formulation of specific objectives Rationale and evidence for omitting major challenges or needs The evaluators may recommend complementary analyses

Coherence Internal coherence External coherence The evaluators should assess: Relationship between specific objectives Coordination mechanisms in multifund programmes External coherence The evaluators should examine: Analysis of programme contribution to other strategies/programmes and of the influence of these on the expected results of the programme Planned integrated territorial approaches

Horizontal principles The evaluators should assess the following elements under each principle (Art. 87(3) CPR): Equality between men an women Arrangements to integrate the gender perspective at the level of the programme and operations Contribution to its promotion Anti-discrimination Specific actions in preparation and implementation of programme Sustainable development Specific actions in the selection of operations Opinion of national equality bodies

Linkage between outputs and results Clear intervention logic: condition for good programming and evaluations! Recommendation: use a logical framework for each priority axis or investment priority The evaluator should examine: - What is the expected change? How will outputs contribute to intended results? Will the proposed actions effectively lead to these outputs? What other factors could influence the expected results? Would evidence suggest other approaches?

Linkage between outputs and results Are the planned form of support the most effective? What is the rationale? (grants, prizes, repayable assistance and financial instruments and a combination: Art. 56) Will the actions effectively meet the needs of specific territories or target groups?

Indicators monitoring and evaluation Relevance and clarity of proposed indicators Baseline and target values Suitability of milestones Administrative capacity Data collection Evaluation Consistency of financial allocation Contribution to Europe 2020 Q&A

Indicators, monitoring and evaluation Art. 24 CPR - For each priority axis, indicators to assess progress towards achievement of objectives - Basis for monitoring, evaluation, review of performance Increased focus on results: role of well designed indicators and evaluations New focus on territorial integrated approaches: reflected in monitoring and evaluations where relevant Common indicators compulsory

Indicators, monitoring and evaluation Relevance of indicators Output indicators reflect the main operations Result indicators are responsive to the policy (values evolve with development of operations) Result indicators reflect the most important intended change Common indicators - ERDF: used where relevant to operations and specific objectives / ESF: monitored under all investment priorities Results may be close to the intervention, short or long term: depends on the intervention logic, on the intended change

Indicators, monitoring and evaluation   Intervention Output indicator Result indicator ERDF Incubator services for start ups Number of incubated start ups Increase of the share of start ups in regional statistics Increase of employment in young enterprises (3 years after incubation) Immediate result indicator Longer-term result indicator ESF Training of individuals who are inactive, not in education or training Number of supported participants (who are inactive, not in education or training) Number of inactive participants who are newly engaged in job searching upon leaving the intervention Participants who are in employment 6 months after leaving

Indicators, monitoring and evaluation Clarity of indicators Unequivoqual title, clear definition Normative interpretation Robustness and transparence Robustness: reliability of data source, representativeness of samples (survey) Data sources for baselines and target values + definitions are publicly available See ex ante conditionality (Annex IV CPR)

Indicators, monitoring and evaluation Baseline and target values Availability of baseline data for result indicators? latest available data (ERDF and ESF Regulations) advice on sources and methods to collect data when necessary, or research to establish baselines Realistic targets? Compared to actions and form of support, financial allocations For output indicators, compared to references (previous programmes or sectoral norms) For result indicators, taking into account other factors

Indicators, monitoring and evaluation Performance framework: a subset of indicators for each priority axis Suitability of milestones Do they capture essential information on the progress of a priority? Can they be achieved at the review points? Will data be available for 2017/2019 progress reports?

Administrative capacity The ex ante evaluator should assess whether adequate human resources and administrative capacity for the management of the programme are in place Including: Have possible previous bottlenecks at the level of MA and IB been tackled? The evaluator may advice on how to address any issues

Data collection The ex ante evaluator shall assess whether the data collection procedures are suitable. Whether they: Respond to all requirements for monitoring including: is data available on time, in the required form and quality? How will the data be collected and stored? Will administrative data bases be used to reduce admin. burden? Etc. Provide the data needed for evaluations including: data to i.a. assess effectiveness, efficiency and impact? What are the mechanisms to ensure good quality data? Etc.

Evaluation The ex ante evaluator may advice: On possible evaluation needs for on-going evaluations On evaluation methodologies and resulting data needs On setting up the evaluation plan

Consistency of financial allocation The ex ante evaluator shall assess the consistency of the financial allocation on the basis of the identified challenges and needs and the proposed actions and selected forms of support The evaluator may assess whether the different funds are adequately combined and contribute to integrated approaches

Contribution to Europe 2020 Taking account of the proportionality principle and against the background of national and regional needs the ex ante evaluator shall assess the programme’s potential contribution to Europe 2020 in the light of its thematic objectives and investment priorities.

Process Strategic Environmental Assessment Timing Interactive and iterative process Partnership and multi-level governance Evaluation methods and financing Final report and publication An independent process What requirements for evaluators? Q&A

Strategic Environmental Assessment A new link with Ex ante: "shall incorporate, where appropriate, the requirements for SEA…" - "Where appropriate": direct impact on environment "Incorporate": does not mean that the ex ante evaluator has to do SEA "Requirements": environmental report, consultations, information, monitoring How to incorporate SEA: still under discussion

Timing Ex ante evaluation & Partnership contracts PC includes (Art. 14): Summary analysis of ex ante evaluations justifying selection of the thematic objectives and the indicative allocation of Funds Summary of main results expected for each thematic objective Developping ToR … now - PC and OPs adopted Oct.-Dec. 2013 - 6 months negotiations: draft PC & OP April-June 2013 Programme elaboration 6-8 months: starts Aug.-Oct. 2012 Calls for tender launched mid 2012

Interactive and iterative process Work in stages Timing and interaction Steering group 29

Partnership and multilevel governance Art. 5 CPR Strengthening with the objective of: Increasing legitimacy Build on wider expertise and knowledge Ensure collective commitment and shared understanding of expected results The evaluators should review if the relevant stakeholders are involved in the preparation of the programme as well as in its implementation, monitoring and evaluation The evaluators should base their assessment on the Commission Staff Working Document “Elements for a Code of Conduct on Partnership” 30

Evaluation methods and financing Mix of methods: - TBE: literature review, workshops, interviews, focus groups, peer reviews Involve partners in programme design Cost: - number of evaluation questions methods Extra tasks (quantification of baselines, complementary analyses)

Final report and publication Submitted with the OP to the Commission - Main evaluation methods used - Changes and improvements to programme - Final assessement of OP - Strategic Environmental Assessment Executive summary (+ in English) Made available to citizens

An independent process Article 47 CPR: "carried out by experts functionally independent of the authorities responsible for programme implementation" No hierarchical link with the entity responsible for implementation: internal expert may contribute Balance between an interactive/iterative and independent process: critical jugements on the different elements of the programme and its coherence

What requirements for evaluators? Knowledge of EU and national documents Deep knowledge of context Situation of areas or target groups with specific needs Situation of economic sectors - Able to judge quality of socio-economic analysis Experience in evaluations Data needs for future impact evaluations Relevance of indicators within the intervention logic (thematic evaluations)