1. Danutė Burakienė, the Ministry of Finance
Quality and Use of Results of EU Structural Funds Evaluation in Lithuania. Lithuanian standards for evaluation of EU structural funds
Dr. V. Nakrošis, PPMI
Danutė Burakienė, the Ministry of Finance
4 July 2013, Vilnius, Lithuania

2. Structure of the presentation
What are the main challenges for the evaluation of EU Cohesion policy in the programming period ?
How is Lithuania addressing these challenges?
Thematic evaluations of quality and use of results of EU Structural Funds in Lithuania;
Lithuanian standards for evaluation of EU structural funds.

3. Main evaluation challenges for the period 2014-2020
Focus on results: monitoring and evaluation (ex-ante evaluation, evaluation during the programming period and ex-post evaluation), as well as their reinforced link;
Impact evaluation: What is an impact? How can one assess it (counterfactual impact evaluation, theory-based evaluation and contribution analysis)?
Implementation evaluation including participatory evaluation and other evaluation types and methods involving stakeholders;
More rigorous evaluation approaches/methods and greater involvement of stakeholders at the same time!

4. Evaluation planning and capacity building
Multi-annual evaluation plan and evaluation capacity measures (human resources, data gathering, a communication strategy, etc.);
Different evaluation systems and approaches require varied responses to new evaluation challanges;
The need to know both alpha (initial situation: where are we?) and omega (future situation: where are we going?)
Evidence-based approach to evaluation planning and capacity building is needed!

5. Dr. V. Nakrošis, PPMI
Quality and Use of Results of EU Structural Funds Evaluation in Lithuania

6. The main purposes of Lithuanian thematic evaluations
To improve the quality of evaluation of the EU structural funds;
To determine how and to which extent evaluation of the EU structural funds has contributed to improving the use of the EU structural funds;
The thematic evaluations assessed 38 evaluations implemented in the period

7. Overall framework for analysis
Use of evaluation results
Demand for evidence in decision-making
PUBLIC MANAGEMENT SYSTEM
Supply of good-quality evidence
Authorities commissioning evaluations
Evaluation service providers

8. The quality of evaluation reports in Lithuania
Meta-analysis: the overall quality of evaluation reports is good (0.81 out of 1);
Surveys: the quality of the evaluation projects is high/very high according to 90% of evaluation commissioners; evaluation quality is good according to service providers
Statistically significant moderate relationship (p<0.05, Kendall‘s tau-c 0.341) between the results of meta-analysis and the perception of respondents

9. However, the quality of evaluation reports varies according to ...
From 0,39 to 0,96
Responsible institutions
Evaluations commissioned by the Ministry of Finance were assessed more positively
Evaluation budget
Lower-budget evaluations proved to be of higher quality (0,82-0,83 compared to 0,73 for larger projects)
Evaluation timing
Evaluations planned in 2011 received higher scores compared to those from the previous years
Criteria of meta-analysis
From 0,71 to 0,92 (with the quality of intervention logics and recommendations assessed least positively)

10. Supply-side and demand-side factors affecting the quality of evaluations
Supply-side factors: competence of evaluators, good project management and cooperation
better technical specifications, longer project duration, better access to data
Demand-side factors:

11. Good record of implementing recommendations
63% of all recommendations have been successfully implemented, only 11% of them have not been implemented.

12. Why some recommendations have not been implemented?
Insufficient quality of recommendations
A statistically significant moderate relationship between the evaluation quality and the number of implemented recommendations (Spearman‘s rho 0.385, p<0.05).
Lack of political will
Insufficient involvement of senior officials
Other motives for decision-making (ideology, interests, institutions, etc.)
Dissemination and monitoring of recommendations
Lack of ownership, insufficient monitoring of recommendations, lack of pro-active approach by responsible institutions, etc.

13. Evaluation chain and links among its processes

14. Evaluation chain and links among its processes (2)
Statistically significant moderate strong direct correlation between the quality score and the share of the recommendations approved and implemented (Spearman’s rho 0.385, p<0.05);
Influence of the previous evaluation processes accumulates in the last stages, whose “wheel” rotation depends on the performance of other “wheels” in the whole evaluation chain.

15. Lessons for the period 2014-2020
System-level: institutionalisation of evaluation
Organisation-level: quality of Terms of Reference, cooperation, monitoring of recommendations
Individual-level: rigorous evaluation methods and dissemination to decision-makers
Evaluation standards can contribute to achieving those ends!