Bangladesh Joint Country Assistance Evaluation: Assessing Total ODA at the Country Level Presentation to OECD DAC November 2006 Bruce Murray Director General Operations Evaluation Department Asian Development Bank

Very Early Stages Everything said is subject to change.

Benefits of Total ODA Evaluation: Richard Manning’s Challenge Consistent with Paris Declaration: (i) mutual accountability; (ii) partnering; (iii) harmonization; (iv) alignment. Reduced transaction costs for Government Objectivity and legitimacy Broader scope Ability to address issues that are beyond those addressed in evaluations undertaken by one donor October 2005 Paper “The effectiveness of ODA – an evaluation proposal” focuses on the impact of total ODA on policy choice and implementation in a country.

Why Bangladesh as Pilot for Total ODA Evaluation? Many donors active in country 4 major donors have a joint country assistance program (WB; ADB; JBIC; DfID) Some major donors require country evaluations by 2009 Use of SWAps.

Phasing of the Evaluation Phase 1 : joint country evaluation by the World Bank, ADB, JBIC/JICA and DFID (80% of ODA). Phase 2: integrate the 28 donors delivering the other 20% of ODA. Daunting coordination problems. Need for strong information sharing between donors involved in Phases 1 and 2. Mechanism not yet developed.

Issues Requiring Resolution: Involving the Government Involving Government as partner in the evaluation work – Paris Declaration on mutual accountability. Proposal to Government must be an open one, including on approach and methodology (listen to Government’s views) Identify issues that the Government wants addressed (e.g., donor coordination; harmonization; use of country systems) Work with Government evaluation units and local research organizations. Involvement of NGOs that deliver much of the bilateral aid.

Issues Requiring Resolution: Approach and Methodology Identify issues that cannot be addressed in the country assistance evaluations of individual donors. What is the counterfactual? Need for common approach and methodology for the separate building blocks of the evaluation. Alignment with joint Country Strategy and PRSP. Note: Joint Strategy assigns lead donors by sectors and themes. Lead donors evaluate all interventions in the sector/thematic area, regardless of funding (data base being assembled to map available evaluation reports and ongoing operations and strategy).

Issues Requiring Resolution: Study Management Need a strong evaluation management mechanism Composition and role of steering committee – Government, 4 major donors and 1 OECD DAC representative. Coordination among task managers and teams, mission planning, etc. How to balance internal needs of donors for evaluation of their programs and joint evaluation (attribution to total ODA, not individual donors) Reporting – one report on total ODA or do donors also need separate reports on their operations for accountability? Design methodology to be flexible enough to produce single or multiple reports. Logistics, timing and cost sharing of the evaluation

Issues Needing Resolution: Involving other Donors How to involve the donors delivering the other 20% of ODA? 1 representative from the OECD DAC Total ODA task force on the steering group Involvement other donors up front in identifying the evaluation questions. Sharing of information during the sector/thematic evaluations Providing information during for the sector/ thematic evaluations Role of country offices and Headquarters evaluation units.

Proposed Schedule Present to end of 2006Data collection and data sharing among main donors (on going); decision on first sector/thematic evaluations among WB/ADB/JBIC- JICA/DFID. Nomination of steering committee for joint evaluation and lead agencies for sectors/themes. January-April 2007Develop management structure for evaluation. Preparation of approach paper for CAE, one sector/thematic evaluation and for Jamuna Bridge/Access Road /Railway Link joint evaluation April/May 2007Proposal for joint country evaluation submitted to Government; joint mission to Bangladesh to discuss proposal. June 2007Detailed decisions on joint country evaluation, costing and timing and mechanism for coordination with the other donors. July 2007Final decision on allocation of other sector/thematic evaluations to various evaluation partners August-October 2007Preparation of detailed evaluation approach papers for initial sector/thematic evaluations, preparation of TORs, consultant selection, detailed planning for joint missions, etc. November 2007-June 2008Field work July – December 2008Discussion and drafting of individual sector/theme reports January – April 2009Drafting and presentation of synthesis report; Dissemination activities.

Challenges of Joint Evaluations (DAC Guidelines) Donor need for accountability and attribution often takes precedence over joint efforts to assess interrelated impacts of multiple agencies Finding subjects suitable for joint evaluations (program support, multilateral, multidonor funded projects) is difficult Processes for coordinating joint evaluations can be complex and increase the cost and duration of the evaluation

DAC Lessons: Joint Evaluations Joint evaluations only reduce transaction costs if they replace, not add to, individual evaluations Identify key partners in evaluation; involvement of government agencies as evaluators is good practice Agree on the management structure early (broad membership steering committee and smaller management group; mixed approaches, some areas of joint evaluation, others to be delivered separately)

DAC Lessons: Joint Evaluations Key areas of agreement: Plan for dissemination and follow up