IDEAS Global Assembly 2011 Evaluation in Times of Turbulence Amman, April 2011 EVALUATION OF THE IMPLEMENTATION OF THE PARIS DECLARATION ON AID EFFECTIVENESS Development Partner Headquarters Studies Case: the Netherlands Ted Kliest Policy and Operations Evaluation Department Netherlands Ministry of Foreign Affairs

‘Ahead of the Crowd?’ Netherlands Headquarters study Purpose: – Like the other HQ studies: ‘mirror’ the partner country-led evaluations and provide an input for the synthesis report by documenting and assessing how the donor has interpreted and translated the PD commitments into policies and procedures for implementation Study (Evaluation) was to assess specifically: – Commitment & leadership: expressed in policies & strategies – Capacity: expressed in guidelines & procedures and their utility; knowledge of the PD and capacities of staff to implement (HQ and field level = embassies) – Adequacy of incentives for implementation and measures taken to overcome possible disincentives Approach: – Combination of document review, interviews at HQ and questionnaire survey covering Dutch embassies in the partner countries which also conducted a country-level evaluation. Plus: interviews with Dutch Development NGOs. – No field work envisaged since field level was to be covered by country-level evaluations

Organisation of the evaluation study Evaluation conducted by the independent Policy and Operations Evaluation Department (IOB) of MoFA Netherlands Team: IOB evaluator/team leader, IOB researcher and external consultant of Institute of Social Studies, The Hague Reference Group – external members: director of European Centre for Development Policy Management and director of a large Dutch development NGO – internal MoFA members: Senior staff member, Effectiveness and Quality Department and Head, Environment and Water Department. – Reference group reviewed and discussed draft ToR and draft report Two internal IOB Peer Reviewers for additional quality assurance

Strengths and weaknesses Strengths: – Relatively simple approach and methodology – Good response & collaboration at HQ and embassy level – Involving NGOs in study and reference group useful to broaden perspective – Aspects to be covered (commitment, capacity and incentives) useful to describe and assess behavioural change as a precondition for the implementation of PD commitments (not real behaviour – see weaknesses) – Ownership with HQ study conducive for subsequent acceptance of synthesis report of stage 1 of the PD evaluation – Both reports send to Dutch Parliament accompanied by management reaction by Minister enhancing their utility

Strengths and weaknesses Weaknesses – Focus (by necessity) only on the input level – Limited time to conduct the evaluation: hence coverage of limited number of embassies. Consequence: missed out embassies which offered better possibilities to investigate harmonisation in practice (e.g. Zambia, Tanzania and Mozambique) – In hindsight: should have assessed actual Dutch behaviour/practice at field level (questionnaire survey not sufficient) more thoroughly because country level evaluations did not sufficiently ‘mirror’ behaviour of the individual donor / agency (= outputs) – Weaknesses somewhat mitigated by providing information on: preliminary results of an ongoing thematic evaluation of the sector approach in water and environment (several countries); and recent changes in Dutch aid modalities (project aid, sector budget, and general budget support) in partner countries covered by the PD evaluation

Update of the Netherlands Headquarters study The original study was published in early 2008 to serve as an input in the synthesis report of the 1st phase of the evaluation of the Paris Declaration Like all who conducted an HQ study, the Netherlands was requested to update its study This has been done by describing the steps / measures taken following HLF 3 in Accra (= uptake of the Accra Agenda for Action) The update document was finalised in October 2010 and included recent information on aid modalities

Observation on all headquarters level evaluations / updates ToRs individual donor/agency studies (based on a framework ToR) not sufficiently shared. Hence: – Evaluations diverged considerably in scope and coverage: Some included field level investigations others did not Minimum requirements (aspects to be covered) stated in framework ToR not always met – Evaluations also diverged in quality: Reports differed regarding the level of detail (information and findings), in frankness, in analytical level, and in their presentation of (critical) conclusions (and recommendations) – Shortcomings could only be mitigated to some extend by the external Peer Review process These drawbacks were avoided by providing more strict guidelines and offering support to evaluation teams engaged in additional HQ studies. Not all earlier HQ evaluations were updated and updates differed in scope. Hence: – a varied set of observations is available for the final synthesis report of phase 2 of the evaluation