1. Quality Assessment of MFA’s evaluations Rita Tesselaar Policy and operations Evaluation Department Netherlands Ministry of Foreign Affairs

2. Content Purpose and scope Quality criteria for evaluations Assessment method Findings in a nutshell Issues assessment method

3. Purpose and scope Purpose: to provide MFA with information for decision making on actions to improve the quality of evaluations Scope: evaluation product (not only the report)

4. Criteria for evaluations Government regulation on evaluations Evaluation criteria: effectiveness and efficiency Quality criteria: validity, reliability, usefulness (result orientation)

5. Assessment method Indicators for quality criteria validity, reliability and usefulness Rating (4-points scale) Data collection –review of documents –interviews

6. indicators validity Reason and purpose: clear description Context: setting -policy and institutional Evaluation object: clear description, intervention logic, result levels, demarcation Evaluation questions: definition of evaluation criteria, clearness, relevance Methods: specification, use of indicators, explanation of limitations Consistency: analyses, conclusions, lessons

7. Indicators reliability Representativeness: account for sample studied Information sources: identification, verification, triangulation of sources Evaluation team: independence Evaluation process: undisturbed operation Quality control: involvement stakeholders, internal or external monitoring committee

8. Indicators usefulness Relevance of results to its intended use: robustness, unambiguous, timely, realism Completeness: questions answered, results explained Presentation: summary, account of methods and process, distinction between conclusions and recommendations

9. Findings (nutshell) Effectiveness criteria partly addressed –focus on process and outputs and less on effects Efficiency criteria gets little attention Quality is variable Evaluations are largely designed from a future oriented managerial perspective

10. Issues assessment method Differences in significance of indicators Subjective element in rating of indicators