The Estonian experience with ex-ante evaluation – set-up and progress Miryam Vahtra Coordinating specialist for evaluations 16.11.2012

Programming process

Ex-ante evaluation Evaluation will be carried out by Praxis in cooperation with Centre for Pure Development. Evaluation team – leading expert on methodology and evaluation content, process leader, evaluation working groups for each field and working group leaders. Kick-off meeting was on 22.08.2012 Inception Report has been presented, includes specific timeline and process description, agreed methodology and the list of interviewees. Evaluators prepare 3 Interim Reports (Excel table supplemented with text) and Final Evaluation Report. Final Evaluation Report is due 28.06.2013 Public presentation of evaluation results in July or August 2013

Ex-ante evaluation (2) Subject to evaluation are the selected objectives and their solutions for EU financing (I report), formed priority axis and initial financing plan (II report), first draft of the partnership agreement and OP (III report) and final version of the partnership agreement and OP. Basis for the evaluation are the documents approved by the cabinet, reports are due 15 working days after approval. In addition recommendations and problem areas are presented if necessary as they arise to enable prompt reaction and taking them into account as soon as possible.

Ex-ante evaluation (3)– theory based evaluation – sources Evidence-based material: Applied research and policy analyses (pre-analyses for 2014+) Previous evaluation reports (all evaluation reports are public) Statistics (Estonian, EU) Policy guidelines Estonian 2020 strategy, sector development plans EU (http://ec.europa.eu/regional_policy/what/future/index_en.cfm) Recommendations made to Estonia (EU stability pact, OECD) Experience of policy experts Interviews with experts coordinating 2014+ planning, involved partners, officials of the SF implementing systems Opinion leaders, researchers, policy field representatives Participation in the planning related meetings and seminars Output from planning process Evaluators have received the list of selected priorities and their solutions for EU financing, which has been discussed with the ministries and presented to the cabinet on 1st of November. According to progress in the planning process next decision papers will follow.

Ex-ante evaluation (4) - Methodological sheets Before each evaluation phase starts the evaluators present us the methodological sheet used for analyses (specified questions, evidence-based triangulation). Always marked 3 sources whether confirming or negating the decisions made – basis for recommendations. Excel sheets enables to track the sources. Detailed explanation of evaluation questions: Evaluation question Sub problem Relevant analyses Explanation what will be evaluated (specific problem, trend etc.) Validity Is the analyses complete (possible additional data needed)? Relations Preferable alternatives

Ex-ante evaluation (5) - Methodological sheets In each evaluation phase one file per field (economy and entrepreneurship, environment, employment etc) On each sheet one criterion In addition summarized sheet (automatic) Must enable data aggregation and analyses Unitary structure Comments for working team responsible for evaluating specific field Kick-off meetings for each evaluation phase to the members of working team Written instructions for filling up the sheets

Ex-ante evaluation (6) - Methodological sheets Ex-ante evaluation – 1 phase - CONCLUSIONS   Field Explanation On this sheet the aggregated analysis is needed regarding the analyses made on sub problems 1.1 - 1.13. Conclusions made on the level of sub problems will be automatically copied to the first sheet. Evaluation questions must have short and clear answers, which are supported by analyses made on the sub problem level. Sub problem (Relevance, Coherence, Impact) Opinion 1.1 To what extent identified problems are valid? (Relevance) 1.2 Does the analyses lack essential problems? (Relevance) 1.5 To what extent the specified development needs are derived logically from problems and external trends? (Relevance) 1.8 To what extent the specific objectives (solutions) are justified? (Relevance, coherence) 1.9 Regarding the development needs to what extent it has been focused on most relevant objectives? (Relevance, impact)  1.13 What are the preconditions necessary for impact? (Impact)  Evaluation questions 1 Whether and to what extent the set objectives of the OP correspond to Estonian development needs. 2 Whether and to what extent the set objectives contribute to the objectives of Europe 2020? 4 To what extent the proposal of objectives and investment priorities presented to the cabinet are relevant and justified regarding the need to ensure the longer term socio-economic development?

Ex-ante evaluation (7) - Methodological sheets 1.1 To what extent the identified problems are valid (Relevance)   Explanation 8 problems with regard to pre-analyses and development needs. Most field do not have as many problems to point at, the rest must be left unfilled. Each problem must be backed up or negated presenting specific sources at least 2-3. Input Final assessment Argumentation Recommendation Problem 1 Source Confirmation 1 Source of confirmation Confirmation 2 Confirmation 3 Argument against 1 Source for negation Argument against 2 Argument against 3 Assessment Comment

Ex-ante evaluation (8) Evaluators attend 27 meetings where proposed priority axis and indicators are discussed. 3 open seminars for 200 participants in November and February where initial findings are introduced. Before each interim report and final evaluation report validation seminars will be organized with representatives of the ministries and 2014+ planning team, where the initial findings and recommendations are discussed. 4 meetings on discussing findings and recommendations presented in evaluation reports with MA and 2014+ planning team.

Ex-ante evaluation (9) Project meetings every 3 weeks for briefing evaluators on the recent decisions and receiving input for the planning process. Evaluation reports will be commented by MA, 2014+ planning team in the Ministry of Finance and the members of the Evaluation Working Group. Agreed methodology - theory based evaluation, no surveys, policy field practical experience incorporated through interviews with leading experts and attending the discussion meetings during the planning process.

Ex-ante evaluation (10) - Main tasks To evaluate the validity of selected national objectives and their solutions for EU financing. Give recommendations for setting the specific objectives and priority axis as well as defining the intervention logic and developing measures. Evaluate internal coherence of the OP and its relation with other national and EU instruments. Assess the relevance of set indicators and verify baselines for result indicators, evaluate monitoring and control systems, identify future evaluation needs and collection of data needed. Give recommendations for forming the implementation system.

Ex-ante evaluation (11) - Ex-ante evaluation covering Partnership Agreement and OP of CP funds and separate contract for SEA co-ordinated by MoF as well as ex-ante evaluation on RDP co-ordinated by Ministry of Agriculture are running in parallel. Ex-ante evaluation for EMFF OP will be commissioned separately. Thank you! Miryam.Vahtra@fin.ee