1. IPA Funds Monitoring and Evaluation 12-16 December 2011 1 Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına Yönelik Teknik Yardım Technical Assistance on Institutional Building for the Implementation of RCOP in Turkey This project is co-financed by the European Union and the Republic of Turkey Evaluation: management Ankara, 15 December 2011 Laura Trofin (PhD): TAT Non-Key Expert, Monitoring and Evaluation lauratrofin@yahoo.com

2. IPA Funds Monitoring and Evaluation 12-16 December 2011 2 This project is co-financed by the European Union and the Republic of Turkey Presentation Structure Planning an evaluation Costs and evaluation budgets Managing stakeholders Role and functions of steering committees Terms of reference Assessing evaluation tenders Managing the evaluator Quality assessment

3. IPA Funds Monitoring and Evaluation 12-16 December 2011 3 This project is co-financed by the European Union and the Republic of Turkey Planning an evaluation: Timing Issues Evaluations take time 1 year for some evaluations from decision to final report Work back from final report deadline Allow time for procurement process Think about when monitoring data will be available Allow 4 – 6 months for draft report Risk management Things will go wrong Get stakeholders and steering committee to agree timetable

4. IPA Funds Monitoring and Evaluation 12-16 December 2011 4 This project is co-financed by the European Union and the Republic of Turkey Planning an evaluation: Timing Issues

5. IPA Funds Monitoring and Evaluation 12-16 December 2011 5 This project is co-financed by the European Union and the Republic of Turkey Evaluation Plan Multi-annual Annual Types, timing, budget of evaluation Feeds into procurement plan, especially for your TA Makes useful results available when you need it

6. IPA Funds Monitoring and Evaluation 12-16 December 2011 6 This project is co-financed by the European Union and the Republic of Turkey Planning an evaluation: Budgets Evaluations cost money! How much? EU Commission recommended 0.5% of programme budget Evaluation Guide: 1% for routine programmes Up to 10% for innovative, pilot programmes Some examples NDP/CSF IE: €400,000 (292 days) OP IE (average of €236,000, 197 days) Priority level: €163,000 (121 days) Policy level: €2,000,000 (1200 days) CBC IE: €60,000 (80 days), 30,000 (35 mandays)

7. IPA Funds Monitoring and Evaluation 12-16 December 2011 7 This project is co-financed by the European Union and the Republic of Turkey Evaluation Planning: Stakeholder Identification Identify and consult with “stakeholders” The individuals, groups, organisations who have an interest in programme to be evaluated European Commission Finance Ministry Managing Authority, Line Ministry Implementing Agency Interest Groups, NGOs Secure their involvement in and cooperation with project Establish their views on evaluation questions

8. IPA Funds Monitoring and Evaluation 12-16 December 2011 8 This project is co-financed by the European Union and the Republic of Turkey Steering Committee What does it do? Oversees evaluation project from start to finish Key tasks Approving terms of reference Selecting the evaluator Quality control Who should be on it? Not a consultation forum Not all stakeholders, keep it manageable Consider involvement of evaluation expert

9. IPA Funds Monitoring and Evaluation 12-16 December 2011 9 This project is co-financed by the European Union and the Republic of Turkey Terms of reference A critical part of evaluation process Poor TOR lead to poor evaluations! TOR should Explain the evaluation context Ask clear questions Identify the key stakeholders Indicate what information is available and when Say how evaluator will be selected and how project will be managed Steering committee should approve

10. IPA Funds Monitoring and Evaluation 12-16 December 2011 10 This project is co-financed by the European Union and the Republic of Turkey TOR includes Request for tender document includes Introduction (summary programme information) Evaluation context Assumptions and risks Purpose, objectives, questions, methods, stakeholders Description of information sources Timing and organisational issues Tender requirements (socio-economic, similar experience, experts) Selection process and award criteria Contract conditions

11. IPA Funds Monitoring and Evaluation 12-16 December 2011 11 This project is co-financed by the European Union and the Republic of Turkey TOR Structure 1. Set the evaluation context 2. State the evaluation objective 3. Specify detailed questions 4. Issues for evaluation conclusions 5. Issues for recommendations

12. IPA Funds Monitoring and Evaluation 12-16 December 2011 12 This project is co-financed by the European Union and the Republic of Turkey TOR example Step 1: Set the evaluation context middle of the implementation process Step 2: State clear evaluation purpose identify solutions to bottlenecks in the programme implementation State 3: clear evaluation questions (as presented earlier) Why has the programme experienced lower demand for funds under Priority/Measure x?

13. IPA Funds Monitoring and Evaluation 12-16 December 2011 13 This project is co-financed by the European Union and the Republic of Turkey TOR example Step 4: Break evaluation question into analytical tasks or sub-questions – essential for you, evaluator further development Analysis of previous evaluation results Ex-ante Ex-post Review of external developments Review of policy developments Progress to date Management and delivery systems Horizontal issues

14. IPA Funds Monitoring and Evaluation 12-16 December 2011 14 This project is co-financed by the European Union and the Republic of Turkey TOR example Step 5: Summarise the issues on which conclusions and recommendations are required – REINFORCES, not ADDS Example: Contribution of CSF to date and likely impact Validity of strategy Appropriateness of financial resources distribution Efficiency of management structures

15. IPA Funds Monitoring and Evaluation 12-16 December 2011 15 This project is co-financed by the European Union and the Republic of Turkey Assessment of Evaluation Tenders: Selection Criteria 4 main criteria What? =analysis and understanding of TOR (context, purpose, objectives, questions/sub-questions) How? =methodology, timing* Who? =experience and resources allocated How much? =cost Each criterion has same/different weight How quantify criteria? Interviews sometimes used – I recommend

16. IPA Funds Monitoring and Evaluation 12-16 December 2011 16 This project is co-financed by the European Union and the Republic of Turkey Managing the Evaluator Keep in touch with the evaluator Request regular updates Set project milestones Inception report Steering committee meetings Evaluator may need help in Establishing contacts Getting data

17. IPA Funds Monitoring and Evaluation 12-16 December 2011 17 This project is co-financed by the European Union and the Republic of Turkey Quality Assessment How does steering committee assess the quality of the evaluation report? Evalsed: 8 criteria Other but evalsed the basis Use of independent evaluation experts

18. IPA Funds Monitoring and Evaluation 12-16 December 2011 18 This project is co-financed by the European Union and the Republic of Turkey Assessment Quality Control: Output criteriaAssessment Meeting needs as laid out in ToR Relevant scope and coverage Defensible design and methods Reliable data used Sound analysis Credible results that relate to analysis and data Impartial conclusions showing no bias and demonstrating sound judgment Clear report with executive summaries and annexed supportive data

19. IPA Funds Monitoring and Evaluation 12-16 December 2011 19 This project is co-financed by the European Union and the Republic of Turkey Quality Assurance: Process criteria Coherent and evaluable objectives Well drawn terms of reference Sound tender selection process Effective dialogue and feedback throughout evaluation process Adequate information resources available Good management and co-ordination by evaluation team Effective dissemination of reports/outputs to Steering Committee and policy/programme managers Effective dissemination to stakeholders

20. IPA Funds Monitoring and Evaluation 12-16 December 2011 20 This project is co-financed by the European Union and the Republic of Turkey Group exercise 5 Completing the evaluation ToR: adding to type, criteria, methods, questions the following elements: purpose, human, time and financial resources, planning and expected results. THINK ABOUT: Why you need the evaluation When do you need the evaluation results How much money you have Tools to manage/monitor the evaluation process, together with the evaluator and Steering Committee Their timing/your time

21. IPA Funds Monitoring and Evaluation 12-16 December 2011 21 This project is co-financed by the European Union and the Republic of Turkey