World Bank Operations Evaluation Department Presentation at 2 nd Meeting of DAC Network on Development Evaluation Paris, November 9-10, 2004 Country Assistance.

Slides:



Advertisements
Similar presentations
Poverty Reduction Strategies: A tool for implementing the BPOA Linda Van Gelder The World Bank.
Advertisements

The Africa Action Plan An IEG Evaluation CSO Forum April 15, 2011.
1 NORAD/OED Trust Fund for Evaluation Activities Nils Fostvedt Senior Advisor to the Director-General, Operations Evaluation May 26, 2005.
Project Ratings: Connects and Disconnects Soniya Carvalho Lead Evaluation Officer and ICR Review Coordinator, IEGPS Independent Evaluation Group.
Country Program Evaluation (CPE) Methodology Anis Dani March 20, 2013.
1 Professionalising Programme & Project Management Developing programme & project management capacities for UNDP and national counterparts External Briefing.
Progress Toward Impact Overall Performance Study of the GEF Aaron Zazueta GEF Evaluation Office Hanoi, March 10, 2010.
Sabelo Mbokazi Senior Policy Officer HIV/AIDS, TB, Malaria & OID
In Europe, When you ask the VET stakeholders : What does Quality Assurance mean for VET system? You can get the following answer: Quality is not an absolute.
World Bank Support for Social Safety Nets Evaluation Findings of the Independent Evaluation Group Presentation to Civil Society Organizations World Bank.
The French Youth Experimentation Fund (Fonds d’Expérimentation pour la Jeunesse – FEJ) Mathieu Valdenaire (DJEPVA - FEJ) International Workshop “Evidence-based.
The Results Agenda Improving The Way We Do Business E. Gail Richardson OPCS – Results Secretariat April 29, 2009.
IFAD Reform towards a better development effectiveness How can we all do better? Mohamed Béavogui Director, West and Central Africa January 2009.
Quality evaluation and improvement for Internal Audit
Office of the Auditor General of Canada The State of Program Evaluation in the Canadian Federal Government Glenn Wheeler Director, Results Measurement.
Purpose of the Standards
Framework for Planning PARIS21 Future Principal Outcomes Sought, and Possible Targets, for World Community.
Internal Auditing and Outsourcing
Corporate-level Evaluation on Gender Equality and Women’s Empowerment Preliminary Findings 63 rd Session of the Evaluation Committee July 2010.
Quality assurance in IVET in Romania Lucian Voinea Mihai Iacob Otilia Apostu 4 th Project Meeting Prague, 21 st -22 nd October 2010.
USAID’s Experience and Lessons Learned in Approaches used in Monitoring and Evaluating Capacity Building Activities Duane Muller, USAID November 7, 2008.
Ghana Country Programme Evaluation National Roundtable Workshop 2 November Accra, Ghana 1 Independent Office of Evaluation.
Results Focus in MDBs: Where Are We? Philibert Afrika, AfDB February 4, 2004.
Title Consultation on the 7 th replenishment of IFAD’s resources IFAD’s operating model : overall structure and components Consultation on the 7th replenishment.
1 RBM Background Development aid is often provided on a point to point basis with no consistency with countries priorities. Development efforts are often.
SECTOR POLICY SUPPORT PROGRAMMES A new methodology for delivery of EC development assistance. 1.
WORLD BANK: SUPPORT FOR HEALTH SECTOR DEVELOPMENT PROGRAM 1.
IAOD Evaluation Section, the Development Agenda (DA) and Development Oriented Activities Julia Flores Marfetan, Senior Evaluator.
1 Analysing the contributions of fellowships to industrial development November 2010 Johannes Dobinger, UNIDO Evaluation Group.
Assessing The Development Needs of the Statistical System NSDS Workshop, Trinidad and Tobago, July 27-29, 2009 Presented by Barbados.
MAINSTREAMING MONITORING AND EVALUATION IN EDUCATION Can education be effectively managed without an M & E system in place?
African Centre for Statistics United Nations Economic Commission for Africa Addressing Data Discrepancies in MDG Monitoring: The Role of UN Regional Commissions.
Mozambique The United Nations At Work THE UN REFORM In Mozambique DaO Evaluability Study, Report and Recommendations May 2009.
Project Management Learning Program 7-18 May 2012, Mekong Institute, Khon Kaen, Thailand Writing Project Report Multi-Purpose Reporting.
1 Corporate-level Evaluation on Gender Equality and Women’s Empowerment IFAD’s Office of Evaluation Informal Seminar Executive Board – 101st Session 13.
European Commission Joint Evaluation Unit common to EuropeAid, Relex and Development Methodology for Evaluation of Budget support operations at Country.
Independent Evaluation Group World Bank November 11, 2010 Evaluation of Bank Support for Gender and Development.
1 Poverty Analysis and Data Initiative (PADI) Capacity Building Program To Support The Poverty Reduction Strategy Shahid Khandker World Bank Institute.
DEVELOPMENT COOPERATION FRAMEWORK Presentation by Ministry of Finance 10 December 2013.
Impact of evaluations matters IDEAS Conference 2011, Amman “Evidence to Policy: Lessons Learnt from Influential Impact Evaluations” Presenter: Daniel Svoboda,
Tracking national portfolios and assessing results Sub-regional Workshop for GEF Focal Points in West and Central Africa June 2008, Douala, Cameroon.
IFAD Reform towards a better development effectiveness How can we all do better? Mohamed Tounessi Bamba Zoumana Virginia Cameroon Retreat 4-5 November.
A short introduction to the Strengthened Approach to supporting PFM reforms.
STRATEGIC REVIEW OF THAILAND’S INTERNATIONAL DEVELOPMENT COOPERATION Presentation to TICA on Methodology 07/06/13.
IFAD’s new operating model Kevin Cleaver Assistant President, Programme Management Department 8-9 July th Replenishment.
M E T R Monitoring and Evaluation for Timely Responses “Measuring the right things in the right way at the right time” Briefing for Discussion at FEMA,
Validated Self Evaluation of Alcohol and Drug Partnerships Evidencing Implementation: The Quality Principles – Care Inspectorate/The Scottish Government.
Corporate-level Evaluation on IFAD’s Private Sector Development and Partnership Strategy 6 th Special Session of the IFAD Evaluation Committee 9 May 2011.
The World Bank’s Experience with Country Environmental Analysis (CEA) Sub-Regional Workshop for GEF Focal Points Europe and the CIS 7-8 March 2007, Istanbul.
GENERAL APPROACH FOR PHASE II OF THE EVALUATION OF THE PARIS DECLARATION ON AID EFFECTIVENESS Phase II Approach Paper.
With Ticon DCA, Copenhagen DC and Ace Global Evaluation of the International Trade Centre Overview, Key Findings, Conclusions and Recommendations Presentation.
Social Analysis Workshop on Country Analytical Work June 19, 2001 Anis Ahmad Dani World Bank, Social Development Department.
What is a Performance Audit or Performance Auditing?
FAO Turkey Partnership Programme (FTPP) FAO Final Evaluation of the FTPP Summary for FTPP Programming Meeting, 14 December
Developing a Monitoring and Evaluation Framework: Insight into internal stakeholder learnings Beth Ferguson AES Conference Sydney 2 September 2011.
Vinod Thomas, Director-General, Evaluation Third International Roundtable Managing for Development Results, Hanoi February 6, 2007 Evaluation for Stronger.
EVALUATION OF THE SEE SARMa Project. Content Project management structure Internal evaluation External evaluation Evaluation report.
1 Workshop on CEA, SEA and DPL January 18, 2005 Kulsum Ahmed, Team Leader, Institutions and Governance Team, Environment Department.
The Role of the Internal and External Evaluators in Student Assessment Arthur Brown Advisor to the Quality Assurance and Accreditation Project Republic.
OED Perspective on ICR Quality Soniya Carvalho, OED Quality At Entry Course on SFs/CDD April 13, 2005 * Contributions from OED’s ICR Review Panel members.
World Bank Group Impact Evaluations: Relevance and Effectiveness Jeff Tanner Task Team Leader for Impact Evaluations Independent Evaluation Group Based.
Improving Effectiveness and Outcomes for the Poor in Health, Nutrition, and Population An Evaluation of World Bank Group Support Since 1997 Martha Ainsworth,
LDCF/SCCF Annual Evaluation Report 2015
FAME DIRECTOR’S OFFICE
… using an approach to Bank assistance that maximizes the effectiveness of scarce resources Greater emphasis on dialogue and analytical work that builds.
World Bank: An Approach to Results-Based Country-Level Evaluations
63rd Session of the Evaluation Committee July 2010
Taking the STANDARDS Seriously
Assessing the Relevance of Global and Regional Partnership Programs (GRPPs) Chris Gerrard Global Programs Coordinator, IEG November 13,
FAME DIRECTOR’S OFFICE
Presentation transcript:

World Bank Operations Evaluation Department Presentation at 2 nd Meeting of DAC Network on Development Evaluation Paris, November 9-10, 2004 Country Assistance Evaluation at OED: Methodology and Challenges

Operations Roadmap 2 Operations Evaluation Department Outline of Presentation 1.Background 2.Country Assistance Evaluation Methodology 3.Rethinking the Approach 4.Relevant Developments in Country Evaluations at the World Bank

Operations Roadmap 3 Operations Evaluation Department Background uSince FY95, OED has undertaken 70 Country Assistance Evaluations, covering nearly three quarters of the Bank’s operational program. uRoughly two-thirds of Bank country assistance programs have been rated satisfactory. uThe usefulness of country evaluations illustrated by disconnect with aggregate portfolio performance.

Operations Roadmap 4 Operations Evaluation Department Satisfactory Project Outcomes do not automatically translate into Satisfactory Assistance Outcomes… SatisfactoryUnsatisfactory Satisfactory53%33% Unsatisfactory7% Country Portfolio Outcomes Outcomes of Bank Assistance Programs CAE Outcome Ratings and Project Outcome Ratings

Operations Roadmap 5 Operations Evaluation Department OED Methodology

Operations Roadmap 6 Operations Evaluation Department  Country development performance;  Assistance program performance; and  Donor (World Bank) performance The first step is to clarify the object of evaluation…

Operations Roadmap 7 Operations Evaluation Department  Country Assistance Programs are assessed across three dimensions:  Development Impact  Products and Services  Attribution Methodology

Operations Roadmap 8 Operations Evaluation Department  Loans, Credits, and Grants  Analytical and Advisory Services  Aid Coordination and Resource Mobilization Dimension I: Products and Services (Bottom-up)

Operations Roadmap 9 Operations Evaluation Department  Main Country-level Program Objectives are derived from:  The country assistance strategy agreed with the Client.  The evaluator’s ex-post assessment of fundamental development constraints.  Major Ratings Same as in Project Evaluations  Relevance  Efficacy  Efficiency  Outcome  Institutional Development  Sustainability Dimension II: Development Impact (Top-Down)

Operations Roadmap 10 Operations Evaluation Department  Bank Performance  Borrower Performance  Partner Performance  Exogenous Factors Dimension III: Attribution

Operations Roadmap 11 Operations Evaluation Department  Individual donor attribution not possible using our current counterfactuals.  Bank performance currently measured against our best practice corporate standards. Attributing Program Results Is a Key Challenge

Operations Roadmap 12 Operations Evaluation Department  Professional Services  Strategy consistent with key development constraints, and corporate priorities  Quality at entry  Implementation and follow-up  Selectivity  Participation and partnership  Creativity, initiative, and efficiency Bank Performance

Operations Roadmap 13 Operations Evaluation Department  Ownership of assistance program  Support for national and international development priorities (MDGs, national development plan, etc.)  Respect for safeguards Client performance

Operations Roadmap 14 Operations Evaluation Department  Impact on design of assistance program  Impact on implementation of assistance program Aid Partner Performance

Operations Roadmap 15 Operations Evaluation Department  World economic shocks  Events of nature  War/civil disturbances  Other Exogenous Factors

Operations Roadmap 16 Operations Evaluation Department Ongoing Review of Methodology…Staff and Board Views Usage Surveys of Board and Bank Staff revealed: uHigh awareness of CAEs and CAE findings uIncorporation of recommendations uFrequent mention in strategies and Board discussions uBut, u Staff views negative overall…timing, accountability, consistency u Board views more positive, but need to focus more on results and less on compliance

Operations Roadmap 17 Operations Evaluation Department ….External Evaluator’s view Methodology reviewed by independent, external reviewer: uThe approach may place too much emphasis on instruments rather than impact uDifferent sections of CAE appear to lead to different conclusions, ie project outcomes uFocus shifts from country to bank to country—may be confusing uNeed to ensure consistency of terminology

Operations Roadmap 18 Operations Evaluation Department …Other Agencies Methods OED made a brief examination of methods employed by other agencies: uOED has completed the most country evaluations by far uOnly about half the agencies have formal, codified methodologies uOED is only agency to rate; very few attempt to construct counterfactuals or attribute

Operations Roadmap 19 Operations Evaluation Department Contemplated Changes to the Methodology… Based on our ongoing self-assessment, we are considering: uMaking the CAEs more-results based, by organizing by objectives u Establishing what the program attempted to accomplish by objective u Analyzing the product dimension by objective rather than overall u Preparing sub-ratings by objectives, disaggregating the overall rating uChanging attribution to the notion of “plausible association” or Bank contribution.

Operations Roadmap 20 Operations Evaluation Department Other Relevant Developments in the Bank… More institution-wide attention to evaluating the impact of country programs uMainstreaming of Results-based CASs, which will: u Contain a CAS Completion Report (self-evaluation), to be validated by OED u More clearly set out what the country program intended to achieve uThe Quality Assurance Group (internal assessments) is also moving to higher-level assessments, such as country-wide ESW or lending reviews.