Evaluation of the Paris Declaration Phase 2 Presentation by Dorte Kabell Member of the Core team Evaluation of the Implementation of the Paris Declaration at IDEAS Global Assembly Amman, April 2011

Why Evaluate the Paris Declaration? Focuses on what has been achieved and what has not – and why – the key questions at HLF 4 and for the post-PD era Evaluation, with Monitoring, is built into the Declaration itself and reflects its principles. The Accra Agenda for Action called specifically for an evaluation of the implementation and effects of the PD Adds value to the Monitoring Survey and feeds into the High Level Forums in 2008 (phase 1) and 2011 (phase 2 synthesis)

Key Elements A common evaluation framework Country-led country level evaluations Development Partners-led HQ level studies Thematic studies to supplement evaluations Intermediate results fed back to Ministerial and public levels (Accra 2008) Outcome evaluation fed back to Ministerial and public levels (Pusan, Korea 2011)

Building blocks of the Evaluation SYNTHESIS PHASE 1 RESULTS and Results of Monitoring Survey EVALUATION QUESTIONS 3. Development outcomes 2. Process and intermediate outcomes 1. Context COUNTRY STUDIES DONOR STUDIES SUPPLEMENTARY STUDIES

Country level evaluations The utility of the Paris Declaration itself as a tool for aid effectiveness The change of donors’ behaviour in terms of alignment of their systems and procedures to implement the PD commitments The change of partner country behaviour, with ownership as the key entry-point Has the implementation of the Paris Declaration strengthened the contribution of aid to sustainable development results? How?

Donor HQ level evaluations Level of leadership and commitment as expressed in policies and strategies Capacity development as expressed in guidelines, procedures, staff training, resources and delegation of authority (to field level) Conducive incentive systems: RBM, HRD In country evaluations, questions of interest to a particular donor can be added or, for some, ‘mirror questions’

The Evaluation and the Monitoring Survey The two exercises are complementary The Monitoring Survey aims to track progress made against selected indicators – the “how much” or “how far” The evaluation results deepen understanding of progress and challenges in the implementation of the PD, and provide explanations for some of the trends highlighted in the Monitoring Survey The evaluation aims at assessing the results achieved and fundamental reasons, the “why?” or “why not?”

The Evaluation Questions 1.To what extent has the Paris Declaration been implemented in different countries and donor/agency systems? 2.What have the effects been in advancing the specific improvements in aid effectiveness targeted in the Declaration? 3.What contributions can aid effectiveness reforms plausibly be judged to have made to development results?

Why these three questions? Q1: PD in context Aid influenced by PD commitments The Aid Partnership Overall development processes Other international & national influences & forces Q1 Q2 Q3 Q3: Effects on PD on development effectiveness Q2: Effects of PD on aid effectiveness

The matrix was developed to meet the challenge of assessing complex change processes Aimed to bring out the theory of change implicit In the Declaration Included a graphic presentation presented to country teams in workshops where matrix was discussed, explained and refined It served as the spine of the evaluation, a common framework that allowed comparisons for synthethis purposes The Operational Matrix

The Context for Implementing the PD – Complex pathways to change

Paragraphs 3 and 4 of the Declaration: 1.Strengthened national strategies and operational frameworks (3.i) 2.Increased alignment (3.ii) 3.Defined measures and standards (3.vi) 4.Reduced duplication of effort (3.iv) 5.Reformed and simplified policies and procedures (3.v) 6.Increased predictability (4.ii) 7.Sufficient delegation (4.iii) 8.Sufficient integration of global initiatives (4.iv) 9.Increased capacity (4.i) 10.Enhanced accountability (3.iii) 11.Reduced corruption and increased transparency (4.v) The 11 intermediate outcomes

Paragraph 2 of the Declaration: The core principles will increase the impact aid has in: Reducing poverty Reducing inequality Increasing growth Building capacity Accelerating achievement of the MDGs Development results

 Each team had Primary Resource Person on Core Team  Support to interpret and use Operational matrix  Guidance notes on Contracting, Attribution and Contribution, Relationship with Monitoring Survey, Use of evidence, i.a.  Workshops, peer learning, telephone/video- conference support, and face-to face support, written comments on drafts, and team meetings in Bali as a last effort to improve weaker reports  Extranet Support to Evaluation Teams

Recognized Limitations Sampling: Self selection, some major countries and institutions absent, contribution from donors in general insufficient Methodology: Complexity of TOR and the object of the evaluation – a policy statement - openness to interpretation, variation of evaluation instruments, effects of indicators, attribution issues Evidence: Varied strength of analysis and evidence in the reports Capacity: Varying capacity of evaluation teams

Process Challenges  Timely delivery of quality input  Procurement obstacles considerable  Governance and organizational issues led to delays  Using the evaluation proces to reinforce the message  No alignment with monitoring survey  Sheer volume of information  Quality assurance

Methodological Challenges for Synthesis Putting aid and aid reform in real context of development – “plausible contributions” not linear attribution of results. Asking to what extent has the Paris Declaration worked & to what extent could it have worked (evaluating the “programme theory”)? Was PD amenable to the necessary political commitment? (Too “technical”? Vested interests too resistant?) Can it be sustained over time and through changes of government? Is a drive for aid effectiveness disciplines undercut by other resource flows (e.g., non-traditional donors, Climate Change funding, claims of “South-South” exceptionalism)?

The (Missed) Opportunities Missed the mark: Timing with monitoring survey Coverage of donors especially in country studies Using process to reinforce message beyond bureaucrats ? Making its mark: Evidencebased contribution to discussion on progress Raise key issues Build evaluation capacity Wider audience in and post Accra and Pusan ?

Making the Evaluation useful A results-oriented evaluation, for wide dissemination and use :  Relevant, credible and readable reports for the Pusan 2011 High Level Forum;  The process should spur interest and improvement in countries and agencies involved;  Useful to donor and partner country governments (executive and legislatures), practitioners (official, civil society and private sector stakeholders) & interested citizens, in both sets of countries; and  Need to be focused, clear (avoid acronyms and jargon) and on time.

Lessons Learned A participatory approach is essential, as is ensuring independence Need a clear, common framework with robust result logic at the core Dealing with context means dealing with complexity Balancing comprehensiveness with feasibility Consider capacity constraints Both methodological challenge and process challenges were considerable Early involvement of Core team in development of TOR in Phase 2 a strength The electronic platform essential