1. Lessons from Programme Evaluation in Romania First Annual Conference on Evaluation Bucharest 18 February 2008

2. Objectives of Session 2 increase the awareness of participants of the existing programme evaluation in the Romanian public sector through the Interim Evaluation of Phare programmes; and identify lessons from interim evaluation for the Romanian Administration in the further use of evaluation in Public Administration.

3. Approach to Session 2 3 Components. For each component: Short presentations (max 10 minutes) Question and answer/ discussion (max 15 minutes) Conclusion (5 minutes) Session Conclusions (max 15 minutes)

4. The three components Component 1 Short presentation on Phare interim evaluation (origins, position in the Phare Management System, link between IE and monitoring, the actors) Component 2 Short presentation on key lessons learned from IE reports (Sector, Country Summary and Thematic Reports) Component 3 Short presentation on lessons learned for the future use of IE in Romanian Public Administration, (structures, management and quality issues in IE)

5. Supporting Material in the Conference Binder Session Abstract Simplified Guide to Interim Evaluation Introduction to the IE Website www.evaluarephare.ro

6. Component 1: Interim Evaluation is Programme Evaluation in Romania Late 1990s: Results monitoring of Phare – (Monitoring and Evaluation combined) 2001: IE introduced when Phare became the major pre-accession instrument (Separate Interim Evaluation): IE is a management tool IE has very close links to the monitoring system There is a well established group of actors needed to make IE successful

7. Current Interim Evaluation Project 20 sector programme evaluation reports – 10 each for 2007 and 2008. 3 Country Summary Reports for the Joint Monitoring Committee meetings. 3 Three Thematic evaluation reports on: Procurement practices and procedures Review of Results Indicators Phare lessons learned for structural funds 1 Applied Methodologies Report 1 Quality Assurance Review report

8. Effects of the Evaluation process Provides independent assessment of sectors and programmes according to relevance, efficiency, effectiveness, impact and sustainability Provides overall assessments for the Joint Monitoring Committee meetings Alerts the authorities through the “Early Warning” system Supports the monitoring system through recommendations to improve Programme performance Systems for monitoring and control

9. The key elements of the IE system Key evaluation capacity elements A sound methodology Independent evaluation capacity A Central Co-ordinating Unit A separate Quality Assurance Group Wider elements A formal programme monitoring structure Adoption of the programme cycle management approach Support from High Civil Servants and the Political Branch

10. Discussion Questions for Component 1 Is the Phare Interim Evaluation System a good model to be used for Programme Evaluation in Romania? How should the system be (a) established (b) resourced?

11. Component 2: Some Key Lessons from IE Reports Programme objectives need to be aligned to National and Sectoral strategies that are kept up to date – there is not enough attention paid to updating the strategies. Switching to a National Procurement System is a complex task that needs careful planning, a lot of training and ad hoc support – the current procurement system should be simplified. Success comes from investing in people – the Romanian administration needs to strengthen its training and development structures and resources. Monitoring systems should focus more on results than on activities or inputs – Further results based reform is needed. A modern strategic management system (e.g. for Structural Funds) requires that responsibility for budgets and results should be decentralised – there is too much control at the centre of the administration.

12. Discussion Questions for Component 2 Do you agree with the lessons from IE on the previous slide? Who are the decision makers that are responsible for each lesson? How can the programme evaluation process influence the decision makers?

13. Component 3: Future use of Programme Evaluation in the Romanian Administration Lessons from the Interim Evaluation Scheme are: A properly defined and resourced structure for monitoring and evaluation is needed for the Programme Evaluation system to work. Communication of evaluation conclusions and recommendations to decision makers is the most important but difficult aspect of the evaluation process

14. Discussion Questions for Component 3 What is needed for the Programme Evaluation process to be more effective? Is the Ministry of Finance a good place for the Central co- ordination of Programme Evaluation activity? Should all Ministries and Agencies have a programme evaluation capacity? Should all programmes be subjected to programme evaluation? If Yes – at what frequency? If No – how should be programmes to be evaluated be selected?

15. A final question A Programme Evaluation system is expensive to create – So, how should the Value of Programme Evaluation be assessed?