1. The Estonian experience with ex-ante evaluation – set-up and progress
Miryam Vahtra
Coordinating specialist for evaluations

2. Programming process

3. Ex-ante evaluation
Evaluation will be carried out by Praxis in cooperation with Centre for Pure Development.
Evaluation team – leading expert on methodology and evaluation content, process leader, evaluation working groups for each field and working group leaders.
Kick-off meeting was on
Inception Report has been presented, includes specific timeline and process description, agreed methodology and the list of interviewees.
Evaluators prepare 3 Interim Reports (Excel table supplemented with text) and Final Evaluation Report.
Final Evaluation Report is due
Public presentation of evaluation results in July or August 2013

4. Ex-ante evaluation (2)
Subject to evaluation are the selected objectives and their solutions for EU financing (I report), formed priority axis and initial financing plan (II report), first draft of the partnership agreement and OP (III report) and final version of the partnership agreement and OP.
Basis for the evaluation are the documents approved by the cabinet, reports are due 15 working days after approval.
In addition recommendations and problem areas are presented if necessary as they arise to enable prompt reaction and taking them into account as soon as possible.

5. Ex-ante evaluation (3)– theory based evaluation – sources
Evidence-based material:
Applied research and policy analyses (pre-analyses for 2014+)
Previous evaluation reports (all evaluation reports are public)
Statistics (Estonian, EU)
Policy guidelines
Estonian 2020 strategy, sector development plans
EU (
Recommendations made to Estonia (EU stability pact, OECD)
Experience of policy experts
Interviews with experts coordinating planning, involved partners, officials of the SF implementing systems
Opinion leaders, researchers, policy field representatives
Participation in the planning related meetings and seminars
Output from planning process
Evaluators have received the list of selected priorities and their solutions for EU financing, which has been discussed with the ministries and presented to the cabinet on 1st of November. According to progress in the planning process next decision papers will follow.

6. Ex-ante evaluation (4) - Methodological sheets
Before each evaluation phase starts the evaluators present us the methodological sheet used for analyses (specified questions, evidence-based triangulation).
Always marked 3 sources whether confirming or negating the decisions made – basis for recommendations.
Excel sheets enables to track the sources.
Detailed explanation of evaluation questions:
Evaluation question Sub problem Relevant analyses
Explanation what will be evaluated (specific problem, trend etc.)
Validity
Is the analyses complete (possible additional data needed)?
Relations
Preferable alternatives

7. Ex-ante evaluation (5) - Methodological sheets
In each evaluation phase one file per field (economy and entrepreneurship, environment, employment etc)
On each sheet one criterion
In addition summarized sheet (automatic)
Must enable data aggregation and analyses
Unitary structure
Comments for working team responsible for evaluating specific field
Kick-off meetings for each evaluation phase to the members of working team
Written instructions for filling up the sheets

8. Ex-ante evaluation (6) - Methodological sheets
Ex-ante evaluation – 1 phase - CONCLUSIONS
Field
Explanation
On this sheet the aggregated analysis is needed regarding the analyses made on sub problems Conclusions made on the level of sub problems will be automatically copied to the first sheet. Evaluation questions must have short and clear answers, which are supported by analyses made on the sub problem level.
Sub problem (Relevance, Coherence, Impact)
Opinion
1.1
To what extent identified problems are valid? (Relevance)
1.2
Does the analyses lack essential problems? (Relevance)
1.5
To what extent the specified development needs are derived logically from problems and external trends? (Relevance)
1.8
To what extent the specific objectives (solutions) are justified? (Relevance, coherence)
1.9
Regarding the development needs to what extent it has been focused on most relevant objectives? (Relevance, impact) 
1.13
What are the preconditions necessary for impact? (Impact) 
Evaluation questions 1
Whether and to what extent the set objectives of the OP correspond to Estonian development needs. 2
Whether and to what extent the set objectives contribute to the objectives of Europe 2020? 4
To what extent the proposal of objectives and investment priorities presented to the cabinet are relevant and justified regarding the need to ensure the longer term socio-economic development?

9. Ex-ante evaluation (7) - Methodological sheets
1.1
To what extent the identified problems are valid (Relevance)
Explanation
8 problems with regard to pre-analyses and development needs. Most field do not have as many problems to point at, the rest must be left unfilled. Each problem must be backed up or negated presenting specific sources at least 2-3.
Input
Final assessment
Argumentation
Recommendation
Problem 1
Source
Confirmation 1
Source of confirmation
Confirmation 2
Confirmation 3
Argument against 1
Source for negation
Argument against 2
Argument against 3
Assessment
Comment

10. Ex-ante evaluation (8)
Evaluators attend 27 meetings where proposed priority axis and indicators are discussed.
3 open seminars for 200 participants in November and February where initial findings are introduced.
Before each interim report and final evaluation report validation seminars will be organized with representatives of the ministries and planning team, where the initial findings and recommendations are discussed.
4 meetings on discussing findings and recommendations presented in evaluation reports with MA and planning team.

11. Ex-ante evaluation (9)
Project meetings every 3 weeks for briefing evaluators on the recent decisions and receiving input for the planning process.
Evaluation reports will be commented by MA, planning team in the Ministry of Finance and the members of the Evaluation Working Group.
Agreed methodology - theory based evaluation, no surveys, policy field practical experience incorporated through interviews with leading experts and attending the discussion meetings during the planning process.

12. Ex-ante evaluation (10) - Main tasks
To evaluate the validity of selected national objectives and their solutions for EU financing.
Give recommendations for setting the specific objectives and priority axis as well as defining the intervention logic and developing measures.
Evaluate internal coherence of the OP and its relation with other national and EU instruments.
Assess the relevance of set indicators and verify baselines for result indicators, evaluate monitoring and control systems, identify future evaluation needs and collection of data needed.
Give recommendations for forming the implementation system.

13. Ex-ante evaluation (11) -
Ex-ante evaluation covering Partnership Agreement and OP of CP funds and separate contract for SEA co-ordinated by MoF as well as ex-ante evaluation on RDP co-ordinated by Ministry of Agriculture are running in parallel.
Ex-ante evaluation for EMFF OP will be commissioned separately.
Thank you!