1. Guidance document on ex-ante evaluation
15 March 2012
Joint presentation to the ERDF and ESF Evaluation Partnership Meeting

2. Ex ante evaluation: our approach
What is different?
Common Regulation & multi-fund programmes
Stronger focus on results / Europe 2020 strategy
More detailed Article 48
Links with Partnership Contracts, SEA
Our common approach:
Stick to Regulation (article 48 and related articles)-as proposed by Commission
Focus on ex ante evaluators' role
When necessary, give examples of differences between ESF/ERDF

3. Ex ante evaluation Guidance
Programme strategy
Q&A
Indicators, monitoring and evaluation
Consistency of financial allocation
Contribution to Europe 2020
Strategic Environmental Assessment
Evaluation Process

4. Programme strategy Consistency of programme objectives Coherence
Horizontal principles
Linkage between supported actions, expected outputs and results
Q&A

5. Programme strategy – key reference points
Art. 24(1) CPR:
Europe 2020 strategy
Common Strategic Framework
Partnership Contract
Art. 87(2)(a)(i) CPR:
Country specific recommendations
national and regional needs

6. Consistency of programme objectives
Step 1: The ex ante evaluator should assess the challenges and needs identified by the programme.
Step 2: The evaluator should examine if these challenges and needs have been consistently translated into thematic objectives, priorities, investment priorities and corresponding specific objectives.

7. Step 1: Challenges & needs (1)
National & regional programmes: the evaluators should primarily base their assessment on the NRP, country-specific recommendations, European semester analyses
In addition:
National ERDF sectoral programmes: specific sectoral analysis & territorial priorities
Regional programmes: specific regional situation and needs
Specific challenges of sub-regional or functional areas or of specific target groups
Horizontal principles (Art. 7 & 8 CPR)
Justification of specific regional challenges and needs

8. Step 1: Challenges & needs (2)
The ex ante evaluator should assess whether the identified challenges and needs are consistent with the analysis of disparities and development needs in the Partnership Contract.
The evaluators should also base their assessment on the Common Strategic Framework.
They should examine the prioritisation of the identified challenges and needs and suggest reviewing it, if appropriate.

9. Step 2: Programme objectives
thematic objectives
investment priorities
Specific objectives
While appraising the consistent translation of
the identified challenge & needs into programme
objectives, the evaluators should focus on:
Their appropriate weight in the investment
priorities
Precise formulation of specific objectives
Rationale and evidence for omitting major challenges or needs
The evaluators may recommend complementary analyses

10. Coherence Internal coherence External coherence
The evaluators should assess:
Relationship between specific objectives
Coordination mechanisms in multifund programmes
External coherence
The evaluators should examine:
Analysis of programme contribution to other strategies/programmes and of the influence of these on the expected results of the programme
Planned integrated territorial approaches

11. Horizontal principles
The evaluators should assess the following elements under each principle (Art. 87(3) CPR):
Equality between men an women
Arrangements to integrate the gender perspective at the level of the programme and operations
Contribution to its promotion
Anti-discrimination
Specific actions in preparation and
implementation of programme
Sustainable development
Specific actions in the selection of operations
Opinion of national equality bodies

12. Linkage between outputs and results
Clear intervention logic: condition for good programming and evaluations!
Recommendation: use a logical framework for each priority axis or investment priority
The evaluator should examine:
- What is the expected change?
How will outputs contribute to intended results?
Will the proposed actions effectively lead to these outputs?
What other factors could influence the expected results?
Would evidence suggest other approaches?

13. Linkage between outputs and results
Are the planned form of support the most effective? What is the rationale?
(grants, prizes, repayable assistance and financial instruments and a combination: Art. 56)
Will the actions effectively meet the needs of specific territories or target groups?

14. Indicators monitoring and evaluation
Relevance and clarity of proposed indicators
Baseline and target values
Suitability of milestones
Administrative capacity
Data collection
Evaluation
Consistency of financial allocation
Contribution to Europe 2020
Q&A

15. Indicators, monitoring and evaluation
Art. 24 CPR
- For each priority axis, indicators to assess progress towards achievement of objectives
- Basis for monitoring, evaluation, review of performance
Increased focus on results: role of well designed indicators and evaluations
New focus on territorial integrated approaches: reflected in monitoring and evaluations where relevant
Common indicators compulsory

16. Indicators, monitoring and evaluation
Relevance of indicators
Output indicators reflect the main operations
Result indicators are responsive to the policy (values evolve with development of operations)
Result indicators reflect the most important intended change
Common indicators - ERDF: used where relevant to operations and specific objectives / ESF: monitored under all investment priorities
Results may be close to the intervention, short or long term: depends on the intervention logic, on the intended change

17. Indicators, monitoring and evaluation
Intervention
Output indicator
Result indicator
ERDF
Incubator services for start ups
Number of incubated start ups
Increase of the share of start ups in regional statistics
Increase of employment in young enterprises (3 years after incubation)
Immediate result indicator
Longer-term result indicator
ESF
Training of individuals who are inactive, not in education or training
Number of supported participants (who are inactive, not in education or training)
Number of inactive participants who are newly engaged in job searching upon leaving the intervention
Participants who are in employment 6 months after leaving

18. Indicators, monitoring and evaluation
Clarity of indicators
Unequivoqual title, clear definition
Normative interpretation
Robustness and transparence
Robustness: reliability of data source, representativeness of samples (survey)
Data sources for baselines and target values + definitions are publicly available
See ex ante conditionality (Annex IV CPR)

19. Indicators, monitoring and evaluation
Baseline and target values
Availability of baseline data for result indicators?
latest available data (ERDF and ESF Regulations)
advice on sources and methods to collect data when necessary, or research to establish baselines
Realistic targets?
Compared to actions and form of support, financial allocations
For output indicators, compared to references (previous programmes or sectoral norms)
For result indicators, taking into account other factors

20. Indicators, monitoring and evaluation
Performance framework: a subset of indicators for each priority axis
Suitability of milestones
Do they capture essential information on the progress of a priority?
Can they be achieved at the review points?
Will data be available for 2017/2019 progress reports?

21. Administrative capacity
The ex ante evaluator should assess whether adequate human resources and administrative capacity for the management of the programme are in place
Including: Have possible previous bottlenecks at the level of MA and IB been tackled?
The evaluator may advice on how to address any issues

22. Data collection
The ex ante evaluator shall assess whether the data collection procedures are suitable. Whether they:
Respond to all requirements for monitoring
including: is data available on time, in the required form and quality? How will the data be collected and stored? Will administrative data bases be used to reduce admin. burden? Etc.
Provide the data needed for evaluations
including: data to i.a. assess effectiveness, efficiency and impact? What are the mechanisms to ensure good quality data? Etc.

23. Evaluation The ex ante evaluator may advice:
On possible evaluation needs for on-going evaluations
On evaluation methodologies and resulting data needs
On setting up the evaluation plan

24. Consistency of financial allocation
The ex ante evaluator shall assess the consistency of the financial allocation on the basis of the identified challenges and needs and the proposed actions and selected forms of support
The evaluator may assess whether the different funds are adequately combined and contribute to integrated approaches

25. Contribution to Europe 2020
Taking account of the proportionality principle and against the background of national and regional needs
the ex ante evaluator shall assess the programme’s potential contribution to Europe 2020 in the light of its thematic objectives and investment priorities.

26. Process Strategic Environmental Assessment Timing
Interactive and iterative process
Partnership and multi-level governance
Evaluation methods and financing
Final report and publication
An independent process
What requirements for evaluators?
Q&A

27. Strategic Environmental Assessment
A new link with Ex ante:
"shall incorporate, where appropriate, the requirements for SEA…"
- "Where appropriate": direct impact on environment
"Incorporate": does not mean that the ex ante evaluator has to do SEA
"Requirements": environmental report, consultations, information, monitoring
How to incorporate SEA: still under discussion

28. Timing Ex ante evaluation & Partnership contracts
PC includes (Art. 14):
Summary analysis of ex ante evaluations justifying selection of the thematic objectives and the indicative allocation of Funds
Summary of main results expected for each thematic objective
Developping ToR … now
- PC and OPs adopted Oct.-Dec. 2013
- 6 months negotiations: draft PC & OP April-June 2013
Programme elaboration 6-8 months: starts Aug.-Oct. 2012
Calls for tender launched mid 2012

29. Interactive and iterative process
Work in stages
Timing and interaction
Steering group 29

30. Partnership and multilevel governance
Art. 5 CPR
Strengthening with the objective of:
Increasing legitimacy
Build on wider expertise and knowledge
Ensure collective commitment and shared understanding of expected results
The evaluators should review if the relevant stakeholders are involved in the preparation of the programme as well as in its implementation, monitoring and evaluation
The evaluators should base their assessment on the Commission Staff Working Document “Elements for a Code of Conduct on Partnership” 30

31. Evaluation methods and financing
Mix of methods:
- TBE: literature review, workshops, interviews, focus groups, peer reviews
Involve partners in programme design
Cost:
- number of evaluation questions
methods
Extra tasks (quantification of baselines, complementary analyses)

32. Final report and publication
Submitted with the OP to the Commission
- Main evaluation methods used
- Changes and improvements to programme
- Final assessement of OP
- Strategic Environmental Assessment
Executive summary (+ in English)
Made available to citizens

33. An independent process
Article 47 CPR: "carried out by experts functionally independent of the authorities responsible for programme implementation"
No hierarchical link with the entity responsible for implementation: internal expert may contribute
Balance between an interactive/iterative and independent process: critical jugements on the different elements of the programme and its coherence

34. What requirements for evaluators?
Knowledge of EU and national documents
Deep knowledge of context
Situation of areas or target groups with specific needs
Situation of economic sectors
- Able to judge quality of socio-economic analysis
Experience in evaluations
Data needs for future impact evaluations
Relevance of indicators within the intervention logic (thematic evaluations)