Cheryl Gray, Director Independent Evaluation Group November 11, 2010 Assessing Effectiveness of World Bank Support to Poverty and Social Impact Analyses (PSIAs) Cheryl Gray, Director Independent Evaluation Group November 11, 2010

What is Poverty and Social Impact Analysis (PSIA)? The analysis of the distributional impact of policy reforms on the well-being of different stakeholder groups, with particular focus on the poor and vulnerable Two elements an analysis to determine the distributional impacts a process that engages appropriate stakeholders in policy-making

Why do a PSIA? Supporting the elaboration and implementation of poverty reduction strategies in developing countries to inform country policies Supporting in-country capacity building to contribute to country analytic capacity Informing Bank operations

Evaluation Questions Effect at the country level: What effect have PSIAs had on country policies (including policy debate)? What contribution have PSIAs made to the development of country capacity for policy analysis? Effect within the Bank: What effect have PSIAs had on Bank operations (including strategy and analytical work)?

Evaluation Methods Desk Review of 58 PSIAs 48 Semi-Structured Stakeholder Interviews 8 Country Case Reviews Literature Review Thematic Review of Donor Involvement A stratified random sample of 58 PSIAs was drawn from the universe of 156 PSIAs Stratification was first by Fiscal Year of funding and then by World Bank Network. The sample was significant at a 95% confidence interval at +/-10% margin of error. Country Case Reviews Chosen purposively with due attention to regional, sectoral, and fiscal year coverage   These eight countries were Bangladesh, Cambodia, Ghana, Malawi, Mali, Mozambique, Nicaragua, and Zambia. Within these countries, 12 PSIAs were examined in-depth. They are listed below. The country case reviews involved visits to the countries concerned, except to Cambodia and Mali. In Cambodia and Mali, extensive interviewing of stakeholders was undertaken, significantly more than that for the semi-structured interviews with country stakeholders and Bank staff. Semi-structured telephone interviews were conducted with 47 stakeholders (9 government officials, 4 private sector representatives, 2 NGO staff, 11 academics and researchers, 2 officials from donor agencies, and 19 Bank staff) chosen purposively to include stakeholders familiar with the PSIA process . Semi-structured interviews were conducted with 30 senior Bank staff and managers (e.g., country directors, advisors, etc.) chosen purposively for their knowledge of PSIAs.

Evaluation Challenges No clear policy on PSIAs Identifying PSIAs Focus was on the influence of a PSIA and not on the technical merit Evaluation did not assess the downstream poverty reduction impacts of PSIAs Defining policy influence was challenging Attribution of government policies and decisions to the PSIAs This evaluation covers analytical work identified as PSIAs by the Bank’s PREM and the SDN Anchors. The evaluation does not cover Bank analyses of poverty and social impacts that the anchors did not identify as PSIAs. The anchors identified 156 analyses that incorporated one or more elements of the PSIA approach over the fiscal 2002–07 period. Most of these are freestanding analyses, but some are embedded in the Bank’s other ESW. Accordingly, “PSIA” is used in this evaluation to cover analytical work that incorporates one or more elements of the PSIA approach, regardless of whether this analytical work is freestanding or embedded. During the course of this evaluation, some Bank staff pointed to the existence of PSIA-type work that was not on the PREM/ SDN list but that should be, and others pointed to PSIAs currently on the list that they believed should not be there. The attribution problems involved in doing so would be prohibitive. Nor does the evaluation aim to assess compliance with OP 8.60, which is being done by PREM and by OPCS. Focus is on near-term effects of PSIAs. Longer-term effects can be difficult to trace and are unlikely to be measurable yet for recent PSIAs. However, the evaluation recognizes that a seemingly ineffective PSIA can be powerfully effective in the future. Other measurement problems also exist. A government decision may accord with recommendations in a PSIA, but similar recommendations may have come from other sources. It is then difficult to assess the PSIA’s contribution. Similar problems of attribution apply to decisions or actions within the Bank. The findings of this evaluation should be interpreted bearing these caveats in mind.

Some Key Findings PSIAs have had widely varying objectives, content, and approaches Majority of PSIAs did not have factors needed to be able to influence policy reform in client countries PSIAs have not been able to build analytic capacity within the country More influence on Bank operations – modest impact Even though PSIAs originated out of concern about the impact of reform programs, about one-third of the PSIAs in the portfolio review did not examine well-specified reforms but were more general sector or macroeconomic analyses. • About one-third of the PSIAs explicitly identified beneficiaries or those adversely affected, and about half did not, although they included disaggregated data or results. • About 60 percent of the sampled PSIAs identified the institutions responsible for implementing the reform. • More than half of the sampled PSIAs included some sort of stakeholder participation, but no consultations were mentioned for about two-fifths of the PSIAs. Informing a policy process is not easy and requires the convergence of a number of factors. These include the operational focus of the PSIA, a match between PSIA topic and country priorities, government ownership, engagement of parts of the government that have policy jurisdiction over the areas covered by the PSIA, engagement with appropriate nongovernmen tal stakeholders, timeliness in relation to country decision making processes, sensitivity to the politics of reform, and active dissemination beyond the distribution of reports. One or a few of these factors have been present in most PSIAs, but the presence of a majority of these factors—which has occurred only in some PSIAs—is important for substantial effect. Capacity bullet point. The PSIAs reviewed suggest a negligible contribution to country analytic capacity, on average, with a few positive examples. The main reason for this finding is that most PSIAs have treated this objective as a by-product rather than a core concern, and there has been a tension between the need to provide timely inputs to policy decisions and the longer-term sustained engagement needed to build capacity. Where capacity building has been an explicit objective, it has not always been backed by an appropriate strategy to achieve it. The PSIAs reviewed suggest a moderate effect on Bank operations, on average, with some outstanding examples of success. Key factors that have inhibited PSIA effect on Bank operations include the ambiguity of the PSIA concept; insufficient country director buy-in, resulting in lack of grounding of the PSIA in the country assistance program; and weak engagement between PSIA teams and other operational staff

Recommendations Improve staff understanding of PSIA approach and its use through improved guidance for staff Clarify the operational objectives of each PSIA with regard to its intended effect and tailor the approach to those objectives Improve integration of the PSIA into the Bank’s country assistance program Strengthen PSIA effectiveness through enhanced quality assurance