1. World Bank: An Approach to Results-Based Country-Level Evaluations
Ajay Chhibber Director
Independent Evaluation, World Bank
Incorporating Managing for Development Results in
Monitoring and Evaluation
Bangkok, February 1, 2006

2. Outline of Presentation
Methodological Approach
Challenges
Experience with Joint Evaluations
Expanding the Approach

3. Methodological Approach: Three Basic Steps
Determining the objectives
Assessing the achievement of each objective
Contribution

4. Country Assistance Evaluation Framework
Country Development Issues
Bank Objectives
Objective 1:
Outcomes relative to:
targets, comparators
Objective n:
Instruments:
Lending, Non-Lending
Partnerships
Outcome Rating for Objective 1
Outcome Rating for Objective n
Overall Assessment
Contribution of Bank
Contribution of Borrower
Contribution of Other Partners
Exogenous Events

5. Step 1: Determining the Objectives of the Bank’s Assistance Strategy
Main Country-level Program Objectives are derived from:
The country assistance strategy agreed with the Client
The evaluator’s ex post assessment of fundamental development constraints.

6. Country Assistance Evaluation Framework
What are the main objectives of the Bank program intended to accomplish? Relevant? Selective? Evaluable?
Bank objectives are a subset of country development objective.
Country Development Issues
Bank Objectives
Objective 1:
Outcomes relative to:
targets, comparators
Objective n:
Instruments:
Lending, Non-Lending
Partnerships
Outcome Rating for Objective 1
Outcome Rating for Objective n
Overall Assessment
Contribution of Bank
Contribution of Borrower
Contribution of Other Partners
Exogenous Events

7. Step 2: Organize by Objectives—Products and Services
For each separate objective:
What was the Bank trying to accomplish?
Were the instruments used:
(i) Loans, Credits, and Grants;
(ii) Analytical and Advisory Services; and,
(iii) Partnerships, Aid Coordination and Resource Mobilization—
Coherent? Relevant to achieving the objective?

8. Country Assistance Evaluation Framework
Country Development Issues
Bank Objectives
Objective 1:
Outcomes relative to:
targets, comparators
Objective n:
Instruments:
Lending, Non-Lending
Partnerships
Outcome Rating for Objective 1
Outcome Rating for Objective n
Overall Assessment
Contribution of Bank
Contribution of Borrower
Contribution of Other Partners
Exogenous Events

9. Illustration of CAE Framework: Tunisia
Proposed Performance Indicators
for
Instruments
Actual Progress
Non-lending
Lending
I. Macroeconomic and Structural Reform Performance Indicators
Sustain an annual GDP growth averaging 5-6, up from 4% in 1995.
Keep annual inflation at 3%, compared to an earlier 6%.
Maintain fiscal deficit around 2% of GDP down from 4%.
Increase private investment to 15.4% of GDP, compared to an earlier 12%
Increase private sector banks as percent of bank assets to 60%, up from 30% in 1995.
II. Key Human Development Performance Indicators
Net primary enrollment ratio of 100% (MDG).
Raise persistence rate to grade 9 from 42% to 73% by 2004.
Reduce unemployment rate at 15% by 2002.
Reduce infant mortality rate to 20 per thousand live births by 2002 (MDG).
III. Key Poverty/rural Performance Indicators
Lower poverty incidence below 7 % (MDG).
Increase access of rural population to safe water (MDG).
Private Sector Assessment Update (FY 01);
Social and Structural Review (FY 00);
Strategy for Public
Public Debt Management (FY 02);
Export growth: determinants and prospects (FY95);
Private sector assessment (FY95);
Towards the 21st Century (FY96);
Higher Education: Challenges and Opportunities (FY97)
ECAL I, II, III ($486m); Export Development ($35m); Transport Sector Investment I-II ($87.6m); ($50m); Private Investment Credit Project ($120m); Industry Support Institutions Upgrading Project ($38.7m); ($60m); Economic and Financial Reforms Support Loan Project ($130m); Public Enterprise Reform Loan Project ($130m).
Training and Employment Project II ($60m); Secondary Education Support
Project ($98.3m); Higher Education Restruct. Project I ($75m); Higher Education Restruct. Project II ($50m);
Employment and Training Fund Project ($12m); Education Quality; Improvement Program Project ($99m); Health Sector Loan ($50m); Hospital Restructuring Support Project ($30m); Education and Training Sector Loan ($95m); Population and Family Health Project ($12m).
GDP growth accelerated to 5.6% over
Annual inflation averaged about 3% during
Fiscal deficit hovered around 3% of GDP during
Private investment/GDP averaged 13.5% over
Privatization proceeds/GDP accumulated to 9%.
Enrollment ratio reached 98.2% in 2000.
Completion rates for grades 6 and 7 are 87% and 63.5% respectively in 2001/2002.
Unemployment rate was 14.9% in 2002.
Infant mortality at 26 per thousand in 2002.
Poverty incidence declined to 4%.
Rural population access to safe water reached 55% in 2002

10. Outcomes: Was the objective achieved?
Mauritania CAE: Progress On Infant Mortality.

11. Outcomes: A Second Example
In Brazil, the Bank focused on improving socio-economic outcomes in the North East.

12. Step 3: Overall Outcome of the Country Assistance Program
In aggregate, were the objectives of the program met?
An Example from the Tunisia CAE:

13. Country Assistance Evaluation Framework
Country Development Issues
Bank Objectives
Objective 1:
Outcomes relative to:
targets, comparators
Objective n:
Instruments:
Lending, Non-Lending
Partnerships
Outcome Rating for Objective 1
Outcome Rating for Objective n
Overall Assessment
Contribution of Bank
Contribution of Borrower
Contribution of Other Partners
Exogenous Events

14. Country Assistance Evaluation Experience
Tunisia CAE : Summary
Objective
Indicator
Performance
Stabilization, Growth, and Structural Reform
Stability
Growth
Financial Sector Liberalization
Trade Reform
Private Sector Development
Low inflation
High
Sounder
Progress
Some progress
Human Resource
Development
Education:
Enrollment
Quality
Health:
Mortality Declines
Spending Efficiency
Little Progress
Poverty & Rural
Poverty Reduction
Agricultural Liberalization
Natural Resource Management
Agricultural Support Service
Land Consolidation
Good Progress
Little progress
Overall Outcomes
Outcome
Sustainability
ID Impact
Satisfactory
Likely
Substantial

15. Step 4: Contribution Bank Performance Borrower Performance
Partner Performance
Exogenous Factors

16. Country Assistance Evaluation Framework
Country Development Issues
Bank Objectives
Objective 1:
Outcomes relative to:
targets, comparators
Objective n:
Instruments:
Lending, Non-Lending
Partnerships
Outcome Rating for Objective 1
Outcome Rating for Objective n
Overall Assessment
Contribution of Bank
Contribution of Borrower
Contribution of Other Partners
Exogenous Events

17. Assessing the Bank’s Contribution
Relevance and implementation of the strategy
Design and supervision of the Bank’s lending
Scope, quality, and follow-up of non-lending work
Consistency of Bank’s lending with its non-lending work
Did the Bank work well with partners, was the program complementary and consultative

18. Assessing Other Contributions:
Client Performance: key issues—Ownership of assistance program; Support for national and international development priorities (MDGs, national development plan, etc.); and respect for safeguards
Partner Performance: key issues—Impact on design of assistance program; and Impact on implementation of assistance program
Exogenous Factors: what was the impact on outcomes of (a) World economic shocks, (b) Events of nature, (c) War/civil disturbances, and (d) other exogenous shocks.

19. Challenges to Assessing Country Assistance Effectively:
Clarifying the object of evaluation.
Reaching agreement on counterfactuals.
Attributing program results correctly.

20. Challenges: 1. Distinguishing between…
Country development performance and the outcome of the assistance program performance: Distinguishing country outcomes from outcomes of Bank assistance programs can be especially problematic for large middle-income borrowers. In Brazil, as noted earlier the Bank focused on the poor in the Northeast; and
The outcome of the assistance program performance and Donor (World Bank) performance: Unsatisfactory outcome ratings do not imply poor Bank performance (e.g., Bulgaria; Haiti; Rwanda are examples of this).

21. Challenges: 2. Finding the Right Counterfactuals
Ideal would be to measure Client development with and without assistance program. (Not feasible, so we have to use proxies.)
Client development before and after assistance program.
Client development relative to countries at similar stage of development.
Client performance relative to the development indicators specified in country assistance strategy.

22. Challenges: 3. Attributing Program Results Correctly
Individual donor attribution not possible using our current counterfactuals, as attribution involves establishing a causal link between a change and a specific intervention.
We have moved to a concept of ‘most likely’ association.
This involves: establishing to the degree possible how and to what extent the Bank’s interventions were linked to the results achieved.

23. OED Evaluation Partnerships
CDF—Multi-donor evaluation
PRS and FSAP (ongoing)—joint evaluation with Independent Evaluation Office (IEO) of IMF
Global Program Evaluations
Country Evaluations with other IFIs

24. (Year of evaluation completion)
OED: Multilateral Partnerships in Country Assistance Evaluations
CAE
(Year of evaluation completion)
Partner
Lesotho (2001)
African Development Bank
Rwanda (2004)
Jordan (2003)
Islamic Development Bank
Tunisia (2004)
Kazakhstan (2001)
International Finance Corporation
Multilateral Investment Guarantee Agency, and European Bank for Reconstruction and Development
Peru (2003)
Inter-American Development Bank
Source: OED Working Paper “Partnership in Joint Country Assistance Evaluations: A Review of World Bank Experience,” OED, World Bank, 2005.

25. Expanding the Approach
Rationale
Lessons from Limited Experience
Issues in Extending the Approach

26. Rationale
Donor community moving towards more coherence in assistance programs:
From a ‘conditionality approach’ to a ‘country-ownership’ approach;
From project aid approach to a budget-support/development policy lending approach;
From a separate procedures and policies approach to a more harmonized approach; and,
From a single donor-to-aid recipient approach to more cooperation among donors.

27. Lessons Learned
Can identify overall impact of donor assistance, as well as key constraints, contradictions and gaps
Promote understanding of institutions’ evaluation methods and encourage the use of common evaluation standards, consistent with the broader agenda of harmonization.
Lower transaction costs for aid recipients by reducing the burden of multiple, separate evaluations for client country institutions, but
Take longer to prepare and cost more than anticipated, as differences in organizational mandates and methodologies can impose constraint/delays on joint evaluations.
Delays are likely to be more pronounced when a single document is produced.

28. Extending the CAE Approach: Some Issues
Establishing the Objectives of the Assistance Programs
Key question is: what are the objectives/results/outcomes that donor programs are attempting to accomplish?
Country development program?
CG agenda of reforms?
PRSP?
MDGs?
Essential to defining the organizing principles of the evaluation

29. Extending the CAE Approach: Some Issues
Attribution/Contribution will present many challenges.
Donor programs may reflect many different objectives, e.g. domestic political concerns—how will we measure relevance of donor programs against objectives
Will it be possible to assess the contribution of the individual donors, e.g. this would involve assessing project versus program assistance, as well as assessing in some cases small interventions.
Is it necessary? Can we focus more on learning what has worked and what has not?

30. Extending the CAE Approach: Some Issues
Country participation—how do we manage it?
The active participation of the recipient country seems to be necessary for an evaluation of this nature.
But, what about Independence questions? Inevitably we will be assessing the Government’s (and past Governments?) contribution to outcomes.
Do we use the MOF, a Supreme Audit Agency, or Experts independent of the Government?