DAC Evaluation Quality Standards Workshop, Auckland 6/2 & 7/ Evaluation quality standards in Dutch Development Cooperation Ted Kliest Policy and Operations Evaluation Department Netherlands Ministry of Foreign Affairs

Evaluation quality standards in Dutch Development Cooperation Subjects: 1.Organisation of evaluation: central & decentral 2.Development of quality standards from the mid- 1990s onwards 3.Use of standards to improve evaluation quality (ex- ante) and assess quality of evaluations (ex-post) 4.DAC Evaluation Quality Standards: some observations

1. Organisation of evaluation in Dutch development cooperation MoFA: a dual system of central and decentral evaluations Central level = independent Policy and Operations Evaluation Department (IOB): –Policy, thematic and (country) programme evaluations, and recently impact evaluations –Increase in joint evaluations –Guidelines for IOB-evaluations (= methods and procedures) Decentral level = evaluations commissioned by operational units and embassies (mainly programmes and projects) Dutch development NGOs have own evaluation functions

2. Decentral evaluation at MoFA: the need for standards Starting in the ‘90s, IOB conducted periodic reviews of the quality (and utility) of decentral evaluations Quality criteria developed for these reviews Periodic reviews led to a Guidance for Decentral Evaluations to be applied in the MoFA Guidance includes quality standards & checklist to foster quality evaluations (ex-ante) and assess quality of evaluations (ex-post) Quality standards and checklist used to assess quality of programme evaluations of Dutch NGOs Experience with standards used by IOB when involved in developing the DAC Quality Standards

3. Use of standards checklist to assess evaluation quality Checklist focuses on three main criteria: 1.Validity - did the evaluation measure what was intended to be measured? 2.Reliability - are the evaluation findings dependable? 3.Usability - can the evaluation results be used? It includes a clarification of concepts It includes criteria & indicators for assessment Quality is scored by means of a 4-point rating system:‘poor’, ‘mediocre’, ‘satisfactory’ and ‘good’

Example: application of ‘Validity’ in the evaluation assessment form Validity operationalised in 4 criteria: 1.Problem definition Clarity of the problem definition Translation of problem definition in evaluation questions Description (& definition) of evaluation criteria 2. Evaluation subject Definition and demarcation of the evaluation object Description of the policy context and institutional context of the subject evaluated

Example of Validity in the evaluation assessment form (continued) 3.Policy theory Description of (reconstructed) intervention logic Operationalisation of the measurement of results by means of indicators (distinction between different results levels) 4.Analysis Specification of evaluation methods applied Justification of the applied methods and techniques (including description of the limitations of the evaluation) Data analysis, and interpretation & formulation of findings (= underpinning of the conclusions) Consistency between conclusions and recommendations

Ex-ante and ex-post application of standards to assess evaluation quality Ex-ante –Standards to be used by operational staff to prepare Evaluation Terms of Reference –Standards applied by IOB to advise operational staff on Evaluation Terms of Reference & evaluation process –Standards to be followed by evaluation teams Ex-post –Standards applied by IOB to assess the quality of 1.Decentral (programme) evaluations at MoFA 2.Programme evaluations conducted by Dutch NGOs

4. DAC Evaluation Quality Standards: some observations Standards: a ‘normative framework’ giving guidance to realise ‘good practice’ evaluations –Provide pointers to develop and conduct evaluations –Are a useful background for staff which is not (very) familiar with evaluation and evaluation processes –Provide useful guidance for evaluation teams But…. Not readily applicable as ex-ante instrument to realise quality evaluations or ex-post instrument to assess the quality of evaluation reports

DAC Evaluation Quality Standards: some observations (continued) Why? –Some sub-headings/explanatory texts are/contain assessment criteria - others do not –Qualitative terms/concepts need definition (for those not very familiar with evaluation) –The list of criteria is very long for quality assessment which should focus on crucial elements –If the Standards are to be used to assess evaluation quality (including meta analysis), they should ideally contain a set of indicators and rating criteria

DAC Evaluation Quality Standards: some observations (continued) Further operationalisation of the Standards as ex-ante guiding instrument and ex-post assessment tool needed? –Issue for discussion in break-out group(s) –Possible input: assessment checklist applied in MoFA Netherlands (handout)