Project Ratings: Connects and Disconnects Soniya Carvalho Lead Evaluation Officer and ICR Review Coordinator, IEGPS Independent Evaluation Group
How are Projects Rated at the World Bank? Project ratings are based on evaluation criteria that are harmonized across IEG and OPCS Main ratings are: Outcome, Risk to Development Outcome, Bank Performance, and Borrower Performance Evaluation methodology is objectives-based
What are the Harmonized Evaluation Criteria? Outcome = Relevance + Efficacy + Efficiency Risk to Development Outcome = Likelihood of detrimental change x Impact on outcome if that change materializes Bank Performance = Quality at Entry + Quality of Supervision Borrower Performance = Government Performance + Implementing Agency(ies) Performance
Why do ICR and ICR Review Ratings Differ? Reason #1: Disagreement over project objectives or their weighting Objectives in ICR are not taken from PAD/Financing Agreement ICR does not discuss how any poorly articulated objectives have been interpreted “By” and “through” parts of objectives are treated as the intended outcomes Relative weights given to different objectives in arriving at ICR ratings are not made explicit and justified
Why do ICR and ICR Review Ratings Differ (Cont’d) Reason #2: Insufficient evidence on achievement of objectives/efficacy Too much focus on outputs and inputs to the neglect of outcomes Reported Key Performance Indicators do not fully capture achievement of objectives Results chain neglected – attribution/plausible association between outputs and outcomes not adequately explained
Why do ICR and ICR Review Ratings Differ? (Cont’d) Reason #3: Differences in treatment of country circumstances In rating Project Outcome, IEG makes no allowance for difficult country context; objectives and design are supposed to take that into account However, difficult country contexts do factor into the Bank Performance rating Project Outcome rating is based on results not effort
Why do ICR and ICR Review Ratings Differ (Cont’d) Reason #4: Lack of familiarity with the ratings ICR rates project outcome based only on achievement of objectives, neglecting relevance and efficiency Relevance in ICR may ignore current relevance Harmonized criteria for deriving Bank Performance from its two constituent elements not observed (Ditto for Borrower Performance)
How can the disconnect be reduced? Ensure clarity about the objectives being used as the benchmark for evaluation Ensure that the ICR contains evidence on the entire results chain, from outputs to outcomes, including for institutional objectives Explicitly discuss attribution/plausible association, providing contextual information on overlapping government or other donor activities in the sector
How can the disconnect be reduced? (Cont’d) Discuss the relevance of project outcomes to the current CAS Acknowledge gaps in achieving the intended outcomes, drawing insightful lessons Acknowledge gaps in M&E, indicating proxy information used to underpin the ICR’s judgments
Even an unsatisfactory project with poor M&E can win an IEG Good Practice ICR Quality Award! Thank you