OED Perspective on ICR Quality Soniya Carvalho, OED Quality At Entry Course on SFs/CDD April 13, 2005 * Contributions from OED’s ICR Review Panel members are gratefully acknowledged.

Objectives of the Presentation To help improve the quality of ICRs. To help understand the main reasons for the disconnect between Regional and OED ratings. To demystify OED’s ICR Review process.

OED’s ICR Review Process OED reviews a 100 percent of all ICRs produced by the Bank. OED’s ICR Review constitutes an independent evaluation of the Bank’s self- evaluation of the project. The ICR Review is based primarily on information provided by the Region in the SAR/PAD, project files, and ICR. (continued)

OED’s ICR Review Process (continued….) The main goals of an ICR Review (like those of other OED evaluations) are to establish accountability and to promote lesson learning.

OED’s Rating Criteria OED’s Outcome rating measures: Relevance: To what extent are objectives consistent with the borrower’s and Bank’s strategy? Efficacy: To what extent are objectives met taking into account their relative importance? Efficiency: To what extent is the project expected to achieve higher return than opportunity cost of capital or least cost compared to alternatives? (continued)

OED’s Rating Criteria (continued….) OED’s other ratings are Sustainability, Institutional Development Impact, Bank Performance, and Borrower Performance. OED and OPCS are currently harmonizing rating criteria so the same definitions and rating scales will be used in ICRs and ICR Reviews.

Questions to Address in the ICR in Assessing Relevance Were the project objectives and proposed activities the “right things to focus on” given the country’s development priorities? Was the project sufficiently focused on long-term development impact and institutional objectives? (continued)

Questions to Address in the ICR in Assessing Relevance (continued….) Did the project rationale adequately address the strategic role of the project within the broader country, poverty, and sector strategies? Was the project the best instrument for achieving the intended development impact in the particular institutional context? (continued)

Questions to Address in the ICR in Assessing Relevance (continued….) Were the activities to be supported by the project relevant to community priorities?

Questions to Address in the ICR in Assessing Efficacy Were the objectives stated in the PAD/DCA well-defined? If not, which interpretation of them is most justified and why? If there were multiple objectives, what is their relative importance? (continued)

Questions to Address in the ICR in Assessing Efficacy (continued….) To what extent were the project’s physical and non-physical objectives met and project targets achieved? What is the attributable connection between what the project did and what actually happened?

Questions to Address in the ICR in Assessing Efficiency To what extent did the project achieve (or is expected to achieve) higher return than the opportunity cost of capital? To what extent did the project provide (or is expected to provide) benefits at least cost compared to alternatives? What was the period between the project investment and the attributable benefit stream (timeliness)?

Disconnect in Outcome Ratings Reason # 1: Insufficient evidence on outcomes One of the main causes of disagreement on ratings between OED and the Regions is insufficient information on outcomes in the ICRs. OED focuses on development outcomes whereas PSRs and ICRs often focus too much on outputs and inputs. (continued)

Disconnect in Outcome Ratings (continued….) Many ICRs disregard attribution/plausible association. Where outcome data have not been presented, OED at least looks for output data and evidence of links between inputs, outputs, and outcomes.

Disconnect in Outcome Ratings Reason #2: Disagreement over project objectives or their weighting  Objectives are sometimes stated in broad terms leaving wide margins for different interpretations.  Sometimes objectives stated in the ICR are different from those in the PAD or Loan Agreement.  Quite often the relative importance of the different objectives is not indicated. (continued)

Disconnect in Outcome Ratings (continued….)  Where it is unavoidable, OED has to infer the objectives or their weighting.  Some ICRs still assess efficacy against completion of components as opposed to achievement of objectives.  OED finds that even in projects where poverty reduction is an objective, the ICR often does not provide evidence on poverty impact.

Disconnect in Outcome Ratings Reason # 3: Differences in treatment of country circumstances OED is sometimes criticized for being too unrealistic because it disregards weak country capacity or a war-ravaged country environment in rating a project’s outcome. OED assumes that a project’s objectives and design have taken into account the country context. OED does make an allowance for the country context in its Bank Performance rating. (continued)

Main Reasons for the Disconnect in ECA Source: ECA Region Quality at Exit Review: Analysis of Recent Exits with Disconnects, FY ECA Region. World Bank, 2003.

How to Reduce the Disconnect Clearly define project objectives and link to measurable indicators. Do not ignore process objectives, e.g., institutional strengthening. Follow development objectives through PSRs/ICR. Indicate weighting or prioritize for multiple objectives. (continued)

How to Reduce the Disconnect (continued….) Present evidence of attribution (plausible association, counterfactual etc). Refer back to ICR guidelines. Discuss draft ratings with experienced staff—not interested parties—to test the ratings.

OED’s ICR Quality Criteria Quality of evidence Quality of analysis Lessons based on evidence and analysis Internal consistency Consistency with ICR guidelines Conciseness (continued)

Good Practice ICR Thailand Lam Takhong Pump Storage Project ICR

Tips for Writing Good ICRs Keep the end in view (OED’s ICR Review Form!). Have serious internal reviews before submitting the ICR. Make the ICR results and outcome-driven— not an implementation narrative. Be consistent—make sure ratings match the text. (continued)

Tips for Writing Good ICRs (continued….) Triangulate all pieces of evidence to substantiate the ratings. Do not neglect the achievement of non-physical and process objectives. Use M&E data to demonstrate attributable connection between what the project did and what actually happened. Do not neglect Annex I of the ICR. Thank You.