Danutė Burakienė, the Ministry of Finance Quality and Use of Results of EU Structural Funds Evaluation in Lithuania. Lithuanian standards for evaluation of EU structural funds Dr. V. Nakrošis, PPMI Danutė Burakienė, the Ministry of Finance 4 July 2013, Vilnius, Lithuania

Structure of the presentation What are the main challenges for the evaluation of EU Cohesion policy in the programming period 2014-2020? How is Lithuania addressing these challenges? Thematic evaluations of quality and use of results of EU Structural Funds in Lithuania; Lithuanian standards for evaluation of EU structural funds.

Main evaluation challenges for the period 2014-2020 Focus on results: monitoring and evaluation (ex-ante evaluation, evaluation during the programming period and ex-post evaluation), as well as their reinforced link; Impact evaluation: What is an impact? How can one assess it (counterfactual impact evaluation, theory-based evaluation and contribution analysis)? Implementation evaluation including participatory evaluation and other evaluation types and methods involving stakeholders; More rigorous evaluation approaches/methods and greater involvement of stakeholders at the same time!

Evaluation planning and capacity building Multi-annual evaluation plan and evaluation capacity measures (human resources, data gathering, a communication strategy, etc.); Different evaluation systems and approaches require varied responses to new evaluation challanges; The need to know both alpha (initial situation: where are we?) and omega (future situation: where are we going?) Evidence-based approach to evaluation planning and capacity building is needed!

Dr. V. Nakrošis, PPMI Quality and Use of Results of EU Structural Funds Evaluation in Lithuania

The main purposes of Lithuanian thematic evaluations To improve the quality of evaluation of the EU structural funds; To determine how and to which extent evaluation of the EU structural funds has contributed to improving the use of the EU structural funds; The thematic evaluations assessed 38 evaluations implemented in the period 2008-2012.

Overall framework for analysis Use of evaluation results Demand for evidence in decision-making PUBLIC MANAGEMENT SYSTEM Supply of good-quality evidence   Authorities commissioning evaluations Evaluation service providers

The quality of evaluation reports in Lithuania Meta-analysis: the overall quality of evaluation reports is good (0.81 out of 1); Surveys: the quality of the evaluation projects is high/very high according to 90% of evaluation commissioners; evaluation quality is good according to service providers Statistically significant moderate relationship (p<0.05, Kendall‘s tau-c 0.341) between the results of meta-analysis and the perception of respondents

However, the quality of evaluation reports varies according to ... From 0,39 to 0,96 Responsible institutions Evaluations commissioned by the Ministry of Finance were assessed more positively Evaluation budget Lower-budget evaluations proved to be of higher quality (0,82-0,83 compared to 0,73 for larger projects) Evaluation timing Evaluations planned in 2011 received higher scores compared to those from the previous years Criteria of meta-analysis From 0,71 to 0,92 (with the quality of intervention logics and recommendations assessed least positively)

Supply-side and demand-side factors affecting the quality of evaluations Supply-side factors: competence of evaluators, good project management and cooperation better technical specifications, longer project duration, better access to data Demand-side factors:

Good record of implementing recommendations 63% of all recommendations have been successfully implemented, only 11% of them have not been implemented.

Why some recommendations have not been implemented? Insufficient quality of recommendations A statistically significant moderate relationship between the evaluation quality and the number of implemented recommendations (Spearman‘s rho 0.385, p<0.05). Lack of political will Insufficient involvement of senior officials Other motives for decision-making (ideology, interests, institutions, etc.) Dissemination and monitoring of recommendations Lack of ownership, insufficient monitoring of recommendations, lack of pro-active approach by responsible institutions, etc.

Evaluation chain and links among its processes

Evaluation chain and links among its processes (2) Statistically significant moderate strong direct correlation between the quality score and the share of the recommendations approved and implemented (Spearman’s rho 0.385, p<0.05); Influence of the previous evaluation processes accumulates in the last stages, whose “wheel” rotation depends on the performance of other “wheels” in the whole evaluation chain.

Lessons for the period 2014-2020 System-level: institutionalisation of evaluation Organisation-level: quality of Terms of Reference, cooperation, monitoring of recommendations Individual-level: rigorous evaluation methods and dissemination to decision-makers Evaluation standards can contribute to achieving those ends!