1. American Evaluation Association Anaheim, 5 November 2011
EVALUATION OF THE IMPLEMENTATION OF
THE PARIS DECLARATION ON AID EFFECTIVENESS
Development Partner Headquarters Studies
Case: the Netherlands
Ted Kliest
Policy and Operations Evaluation Department
Netherlands Ministry of Foreign Affairs

2. ‘Ahead of the Crowd?’ Netherlands Headquarters study
Purpose:
Like the other HQ studies: ‘mirror’ the partner country-led evaluations and provide an input for the synthesis report by documenting and assessing how the donor has interpreted and translated the PD commitments into policies and procedures for implementation
Study (Evaluation) was to assess specifically:
Commitment & leadership: expressed in policies & strategies
Capacity: expressed in guidelines & procedures and their utility; knowledge of the PD and capacities of staff to implement (HQ and field level = embassies)
Adequacy of incentives for implementation and measures taken to overcome possible disincentives
Approach:
Combination of document review, interviews at HQ and questionnaire survey covering Dutch embassies in the partner countries which also conducted a country-level evaluation. Plus: interviews with Dutch Development NGOs.
No field work envisaged since field level was to be covered by country-level evaluations

3. Organisation of the evaluation study
Evaluation conducted by the independent Policy and Operations Evaluation Department (IOB) of MoFA Netherlands
Team: IOB evaluator/team leader, IOB researcher and external consultant of Institute of Social Studies, The Hague
Reference Group
external members: director of European Centre for Development Policy Management and director of a large Dutch development NGO
internal MoFA members: Senior staff member, Effectiveness and Quality Department and Head, Environment and Water Department.
Reference group reviewed and discussed draft ToR and draft report
Two internal IOB Peer Reviewers for additional quality assurance

4. Strengths and weaknesses
Relatively simple approach and methodology
Good response & collaboration at HQ and embassy level
Involving NGOs in study and reference group useful to broaden perspective
Aspects to be covered (commitment, capacity and incentives) useful to describe and assess behavioural change as a precondition for the implementation of PD commitments (not real behaviour – see weaknesses)
Ownership with HQ study conducive for subsequent acceptance of synthesis report of stage 1 of the PD evaluation
Both reports send to Dutch Parliament accompanied by management reaction by Minister enhancing their utility

5. Strengths and weaknesses
Focus (by necessity) only on the input level
Limited time to conduct the evaluation: hence coverage of limited number of embassies. Consequence: missed out embassies which offered better possibilities to investigate harmonisation in practice (e.g. Zambia, Tanzania and Mozambique)
In hindsight: should have assessed actual Dutch behaviour/practice at field level (questionnaire survey not sufficient) more thoroughly because country level evaluations did not sufficiently ‘mirror’ behaviour of the individual donor / agency (= outputs)
Weaknesses somewhat mitigated by providing information on:
preliminary results of an ongoing thematic evaluation of the sector approach in water and environment (several countries); and
recent changes in Dutch aid modalities (project aid, sector budget, and general budget support) in partner countries covered by the PD evaluation

6. Update of the Netherlands Headquarters study
The original study was published in early 2008 to serve as an input in the synthesis report of the 1st phase of the evaluation of the Paris Declaration
Like all who conducted an HQ study, the Netherlands was requested to update its study
This has been done by describing the steps / measures taken following HLF 3 in Accra (= uptake of the Accra Agenda for Action)
The update document was finalised in October 2010 and included recent information on aid modalities

7. Observation on all headquarters level evaluations / updates
ToRs individual donor/agency studies (based on a framework ToR) not sufficiently shared. Hence:
Evaluations diverged considerably in scope and coverage:
Some included field level investigations others did not
Minimum requirements (aspects to be covered) stated in framework ToR not always met
Evaluations also diverged in quality:
Reports differed regarding the level of detail (information and findings), in frankness, in analytical level, and in their presentation of (critical) conclusions (and recommendations)
Shortcomings could only be mitigated to some extend by the external Peer Review process
These drawbacks were avoided by providing more strict guidelines and offering support to evaluation teams engaged in additional HQ studies.
Not all earlier HQ evaluations were updated and updates differed in scope. Hence:
a varied set of observations was available for the Synthesis of the results of the Paris Declaration Evaluation and its constituent studies/evaluations