World Bank: An Approach to Results-Based Country-Level Evaluations Ajay Chhibber Director Independent Evaluation, World Bank Incorporating Managing for Development Results in Monitoring and Evaluation Bangkok, February 1, 2006

Outline of Presentation Methodological Approach Challenges Experience with Joint Evaluations Expanding the Approach

Methodological Approach: Three Basic Steps Determining the objectives Assessing the achievement of each objective Contribution

Country Assistance Evaluation Framework Country Development Issues Bank Objectives Objective 1: Outcomes relative to: targets, comparators Objective n: Instruments: Lending, Non-Lending Partnerships Outcome Rating for Objective 1 Outcome Rating for Objective n Overall Assessment Contribution of Bank Contribution of Borrower Contribution of Other Partners Exogenous Events

Step 1: Determining the Objectives of the Bank’s Assistance Strategy Main Country-level Program Objectives are derived from: The country assistance strategy agreed with the Client The evaluator’s ex post assessment of fundamental development constraints.

Country Assistance Evaluation Framework What are the main objectives of the Bank program intended to accomplish? Relevant? Selective? Evaluable? Bank objectives are a subset of country development objective. Country Development Issues Bank Objectives Objective 1: Outcomes relative to: targets, comparators Objective n: Instruments: Lending, Non-Lending Partnerships Outcome Rating for Objective 1 Outcome Rating for Objective n Overall Assessment Contribution of Bank Contribution of Borrower Contribution of Other Partners Exogenous Events

Step 2: Organize by Objectives—Products and Services For each separate objective: What was the Bank trying to accomplish? Were the instruments used: (i) Loans, Credits, and Grants; (ii) Analytical and Advisory Services; and, (iii) Partnerships, Aid Coordination and Resource Mobilization— Coherent? Relevant to achieving the objective?

Country Assistance Evaluation Framework Country Development Issues Bank Objectives Objective 1: Outcomes relative to: targets, comparators Objective n: Instruments: Lending, Non-Lending Partnerships Outcome Rating for Objective 1 Outcome Rating for Objective n Overall Assessment Contribution of Bank Contribution of Borrower Contribution of Other Partners Exogenous Events

Illustration of CAE Framework: Tunisia Proposed Performance Indicators for 1996-2002 Instruments Actual Progress Non-lending Lending I. Macroeconomic and Structural Reform Performance Indicators Sustain an annual GDP growth averaging 5-6, up from 4% in 1995. Keep annual inflation at 3%, compared to an earlier 6%. Maintain fiscal deficit around 2% of GDP down from 4%. Increase private investment to 15.4% of GDP, compared to an earlier 12% 1990-96. Increase private sector banks as percent of bank assets to 60%, up from 30% in 1995. II. Key Human Development Performance Indicators Net primary enrollment ratio of 100% (MDG). Raise persistence rate to grade 9 from 42% to 73% by 2004. Reduce unemployment rate at 15% by 2002. Reduce infant mortality rate to 20 per thousand live births by 2002 (MDG). III. Key Poverty/rural Performance Indicators Lower poverty incidence below 7 % (MDG). Increase access of rural population to safe water (MDG). Private Sector Assessment Update (FY 01); Social and Structural Review (FY 00); Strategy for Public Public Debt Management (FY 02); Export growth: determinants and prospects (FY95); Private sector assessment (FY95); Towards the 21st Century (FY96); Higher Education: Challenges and Opportunities (FY97) ECAL I, II, III ($486m); Export Development ($35m); Transport Sector Investment I-II ($87.6m); ($50m); Private Investment Credit Project ($120m); Industry Support Institutions Upgrading Project ($38.7m); ($60m); Economic and Financial Reforms Support Loan Project ($130m); Public Enterprise Reform Loan Project ($130m). Training and Employment Project II ($60m); Secondary Education Support Project ($98.3m); Higher Education Restruct. Project I ($75m); Higher Education Restruct. Project II ($50m); Employment and Training Fund Project ($12m); Education Quality; Improvement Program Project ($99m); Health Sector Loan ($50m); Hospital Restructuring Support Project ($30m); Education and Training Sector Loan ($95m); Population and Family Health Project ($12m). GDP growth accelerated to 5.6% over 1996-02. Annual inflation averaged about 3% during 1996-02. Fiscal deficit hovered around 3% of GDP during 1996-02. Private investment/GDP averaged 13.5% over 1997-02. Privatization proceeds/GDP accumulated to 9%. Enrollment ratio reached 98.2% in 2000. Completion rates for grades 6 and 7 are 87% and 63.5% respectively in 2001/2002. Unemployment rate was 14.9% in 2002. Infant mortality at 26 per thousand in 2002. Poverty incidence declined to 4%. Rural population access to safe water reached 55% in 2002

Outcomes: Was the objective achieved? Mauritania CAE: Progress On Infant Mortality.

Outcomes: A Second Example In Brazil, the Bank focused on improving socio-economic outcomes in the North East.

Step 3: Overall Outcome of the Country Assistance Program In aggregate, were the objectives of the program met? An Example from the Tunisia CAE:

Country Assistance Evaluation Framework Country Development Issues Bank Objectives Objective 1: Outcomes relative to: targets, comparators Objective n: Instruments: Lending, Non-Lending Partnerships Outcome Rating for Objective 1 Outcome Rating for Objective n Overall Assessment Contribution of Bank Contribution of Borrower Contribution of Other Partners Exogenous Events

Country Assistance Evaluation Experience Tunisia CAE : Summary Objective Indicator Performance Stabilization, Growth, and Structural Reform Stability Growth Financial Sector Liberalization Trade Reform Private Sector Development Low inflation High Sounder Progress Some progress Human Resource Development Education: Enrollment Quality Health: Mortality Declines Spending Efficiency Little Progress Poverty & Rural Poverty Reduction Agricultural Liberalization Natural Resource Management Agricultural Support Service Land Consolidation Good Progress Little progress Overall Outcomes Outcome Sustainability ID Impact Satisfactory Likely Substantial

Step 4: Contribution Bank Performance Borrower Performance Partner Performance Exogenous Factors

Country Assistance Evaluation Framework Country Development Issues Bank Objectives Objective 1: Outcomes relative to: targets, comparators Objective n: Instruments: Lending, Non-Lending Partnerships Outcome Rating for Objective 1 Outcome Rating for Objective n Overall Assessment Contribution of Bank Contribution of Borrower Contribution of Other Partners Exogenous Events

Assessing the Bank’s Contribution Relevance and implementation of the strategy Design and supervision of the Bank’s lending Scope, quality, and follow-up of non-lending work Consistency of Bank’s lending with its non-lending work Did the Bank work well with partners, was the program complementary and consultative

Assessing Other Contributions: Client Performance: key issues—Ownership of assistance program; Support for national and international development priorities (MDGs, national development plan, etc.); and respect for safeguards Partner Performance: key issues—Impact on design of assistance program; and Impact on implementation of assistance program Exogenous Factors: what was the impact on outcomes of (a) World economic shocks, (b) Events of nature, (c) War/civil disturbances, and (d) other exogenous shocks.

Challenges to Assessing Country Assistance Effectively: Clarifying the object of evaluation. Reaching agreement on counterfactuals. Attributing program results correctly.

Challenges: 1. Distinguishing between… Country development performance and the outcome of the assistance program performance: Distinguishing country outcomes from outcomes of Bank assistance programs can be especially problematic for large middle-income borrowers. In Brazil, as noted earlier the Bank focused on the poor in the Northeast; and The outcome of the assistance program performance and Donor (World Bank) performance: Unsatisfactory outcome ratings do not imply poor Bank performance (e.g., Bulgaria; Haiti; Rwanda are examples of this).

Challenges: 2. Finding the Right Counterfactuals Ideal would be to measure Client development with and without assistance program. (Not feasible, so we have to use proxies.) Client development before and after assistance program. Client development relative to countries at similar stage of development. Client performance relative to the development indicators specified in country assistance strategy.

Challenges: 3. Attributing Program Results Correctly Individual donor attribution not possible using our current counterfactuals, as attribution involves establishing a causal link between a change and a specific intervention. We have moved to a concept of ‘most likely’ association. This involves: establishing to the degree possible how and to what extent the Bank’s interventions were linked to the results achieved.

OED Evaluation Partnerships CDF—Multi-donor evaluation PRS and FSAP (ongoing)—joint evaluation with Independent Evaluation Office (IEO) of IMF Global Program Evaluations Country Evaluations with other IFIs

(Year of evaluation completion) OED: Multilateral Partnerships in Country Assistance Evaluations CAE (Year of evaluation completion) Partner Lesotho (2001) African Development Bank Rwanda (2004) Jordan (2003) Islamic Development Bank Tunisia (2004) Kazakhstan (2001) International Finance Corporation Multilateral Investment Guarantee Agency, and European Bank for Reconstruction and Development Peru (2003) Inter-American Development Bank Source: OED Working Paper “Partnership in Joint Country Assistance Evaluations: A Review of World Bank Experience,” OED, World Bank, 2005.

Expanding the Approach Rationale Lessons from Limited Experience Issues in Extending the Approach

Rationale Donor community moving towards more coherence in assistance programs: From a ‘conditionality approach’ to a ‘country-ownership’ approach; From project aid approach to a budget-support/development policy lending approach; From a separate procedures and policies approach to a more harmonized approach; and, From a single donor-to-aid recipient approach to more cooperation among donors.

Lessons Learned Can identify overall impact of donor assistance, as well as key constraints, contradictions and gaps Promote understanding of institutions’ evaluation methods and encourage the use of common evaluation standards, consistent with the broader agenda of harmonization. Lower transaction costs for aid recipients by reducing the burden of multiple, separate evaluations for client country institutions, but Take longer to prepare and cost more than anticipated, as differences in organizational mandates and methodologies can impose constraint/delays on joint evaluations. Delays are likely to be more pronounced when a single document is produced.

Extending the CAE Approach: Some Issues Establishing the Objectives of the Assistance Programs Key question is: what are the objectives/results/outcomes that donor programs are attempting to accomplish? Country development program? CG agenda of reforms? PRSP? MDGs? Essential to defining the organizing principles of the evaluation

Extending the CAE Approach: Some Issues Attribution/Contribution will present many challenges. Donor programs may reflect many different objectives, e.g. domestic political concerns—how will we measure relevance of donor programs against objectives Will it be possible to assess the contribution of the individual donors, e.g. this would involve assessing project versus program assistance, as well as assessing in some cases small interventions. Is it necessary? Can we focus more on learning what has worked and what has not?

Extending the CAE Approach: Some Issues Country participation—how do we manage it? The active participation of the recipient country seems to be necessary for an evaluation of this nature. But, what about Independence questions? Inevitably we will be assessing the Government’s (and past Governments?) contribution to outcomes. Do we use the MOF, a Supreme Audit Agency, or Experts independent of the Government?