Ten years of Evaluability at the IDB Yuri Soares, Alejandro Pardo, Veronica Gonzalez and Sixto Aquino Paris, 16 November, 2010

Preliminaries Evaluability is the “ability of an intervention to demonstrate in measurable terms the results it intends to deliver” (IDB, 2002, 2006, 2010). In terms of Development Effectiveness, the Bank’s capacity to manage for results depends to a great extent on having operations that feature the characteristics needed for results to be measured, as well as the understanding of the main factors affecting the process by which they are generated.

IDB Experience The IDB’s Evaluation Office has produced three assessments of project evaluability: 2001, 2005, and These exercises systematically reviewed the universe of IDB projects approved during these years. Thus, the institution now has longitudinal data on project evaluability. The 2005 Evaluability Report recommended that evaluability standards be introduced as a criterion for project approval. This recommendation was adopted and included in the institution’s mandate as part of its Ninth General Capital Increase (2010). The institution is currently moving toward implementing this mandate; and OVE has been instructed to perform evaluability assessments on a yearly basis.

Evaluability Dimensions The evaluability consists of nine dimensions including substantive and formal dimensions. Substantive dimensions are those that assess the proper identification and linkages between the conceptual elements of an intervention. Formal dimensions are those that go to the “classical” measures of evaluability, such as the identification of indicators and baselines. Substantive Diagnostic Objectives Logic Assumptions and risks Formal Output baselines Outcome baselines Output indicators Outcome indicators Monitoring and evaluation

Substantive Dimensions  Diagnosis: Evidence-based identification of the problem and its roots causes.  Objectives: Identification of what project expect to achieve. Objectives must be S.M.A.R.T. (Specific, Measurable, Agreed upon, Realistic, Temporal).  Logic: why this particular intervention and why not others? Causal chain: components  create conditions  produce outputs  achieve outcomes  Risks: Quality of analysis in the identification of assumptions & risks. Risk Evaluation, Follow-up and Mitigation

Formal Dimensions  Outcome indicators : measures of expected results during and/or at end of project  Output Indicators: measure of expected products executed as part of the operation  Indicators must be mutually exclusive, valid, adequate, and reliable  Baselines for outcomes: Ex-ante assessments of conditions expected to change as a result of project  Baselines for outputs: Ex-ante assessments of the goods and services present prior to the project  Monitoring and Evaluation: Identification of systems and resources for data collection.

Protocol To ensure the proper application of the exercise, a protocol was designed and implemented, consisting of three steps:  Write-up of findings. Project assessments are done by peer review. Reviewers meet, discuss the proposed operation, and produce a note reporting on findings in each evaluability dimensions.  Collegial review. The findings of the note are then discussed by a collegial group in the office. This same group reviews all projects to ensure consistency across projects.  Scoring. Once a final draft is produced, the team and collegial group agree on a scoring for each of the dimensions. The scoring scale is a 1-4 scale, with two adequate and two inadequate categories, and is based on a scoring guide.

Principles The method adheres to a series of principles:  Quality. Reviews are done by peer reviews; these peers contain staff with knowledge of the sector and problematic.  Independence. To avoid conflicts of interest, staff who may have had involvement in a project do not participate in the review.  Accountability. A manager oversees the overall exercise and is responsible for its quality.  Consistency. All reviews are validated by the same collegial group, so as to ensure consistency across different project reviews.

Results

An Operations microscope The review tracks the evaluability of the Bank’s principal production function: the design of lending operations. This can provide insights regarding specific problems encountered in projects in order to improve them. For example:  Identify what efforts are needed to ensure that problem situations are accurately dimensioned and ensure that the Bank has evidence regarding the aptness of the intervention models  Identify overlooked risks that can impact the viability of the interventions in order to ensure better risk management.  Address and quantify “classical” evaluability questions, such as if outcome indicators are adequately defined, and sufficient for the scope of the operation’s objectives, and if they have baseline data. They have clear implications for the institution’s monitoring and data collection efforts.

A Institution microscope Our experience with evaluability shows that it can also be a valuable tool to look at how the Bank operates. We have done this by analyzing the link between evaluability trends and:  changes in quality review process,  variations in incentives faced by teams and managers,  impact of the organizational changes in the quality of projects,  allocation of staff and resources, and  fluctuations in lending portfolio

Oversight example: In 2005 and in 2009, the IDB analyzed the functioning of the institution’s quality control function. Findings included that Managers, Bank committees, and peer review instances were not providing sufficient guidance to project teams, and in almost no case were they providing oversight of evaluability-related issues of project design. Also, divisions formally responsible for quality control were mostly absent from the review process. National systems example. In 2005 found evidence that the Institution’s bureaucratic incentives were not aligned with those of borrowers. OVE recommended the use evaluability in order determine the Bank’s value in project design in order to inform to decisions of increased use of national planning systems Risk example. In 2005 and in 2009 the Evaluability assessment looked at how the institution managed risk at the project level. The results found that for sovereign guaranteed operations issues of risk were rarely identified, while for private sector operations risks were always front and center, but were exclusively focused on financial risks responding to the repayment related incentives

Concluding remarks Evaluability provides CEDs with an opportunity to play an important role in assessing and influencing the way development institutions discharge their mandates Evaluability assesses critical elements of the quality of the work of the organizations as related to their capacities for management for results Evaluability can be used as a tool for looking at how institutions organize and for steering steer critical institutional improvement processes