Impact Evaluation of Urban Upgrading Programs Judy Baker, FEU November 19, 2007

Impact Evaluation Program Mainstreaming Impact Evaluations in relevant Bank Projects Cluster of Evaluations to evaluate impact of urban upgrading projects  Afghanistan, Brazil, Colombia, Honduras, India, Indonesia, Iran, Jamaica, Nigeria, Tanzania Part of Development Impact Evaluation (DIME) Program; First SDN cluster Project designs vary, IE designs also vary.  Usually include physical upgrading (water and sanitation, roads, housing improvements, street lighting, etc.)  Sometimes include social components (job training, social infrastructure, crime prevention, community development)  Sometimes include land tenure

Main research questions What are the impacts of upgrading projects?  Welfare (employment, income and consumption)  Land and housing values  Health and Education outcomes  Community (social capital)  Tenure outcomes where feasible Do specific design features have variable impacts? How cost effective are interventions? Targeting outcomes ? Meta Analysis will be done in two parts: Baseline, and IE Results.

Issues in designing the evaluations Identifying appropriate and interested projects (TTLs, Government, Resources) Staffing, Resources Quasi-experimental designs, very difficult to randomize in infrastructure projects Are there existing data sets ? Implementation of baseline needs to be timely Mobility in urban areas tends to be high

Example: Bahia Integrated Urban Upgrading PDO: to reduce urban poverty in a sustainable manner, targeting the poorest and most vulnerable sections of Salvador and strategic cities in Bahia with access to basic services and improved housing and social support services. Investment loan including: Urban Infrastructure (65%) Social Service Delivery (25%) Institutional Strengthening (10%)  M&E (1.2%)

Monitoring and Evaluation Task manager initiated dialogue Drew on pilot project Government had strong commitment to evaluation, relationships already established Counterpart team assigned to preparation of M&E design Approach discussed at stakeholders meeting during project preparation

M&E Design Monitoring and Process Evaluation  Input from MIS, existing administrative data  Reporting carried out quarterly by implementing agency  Participatory monitoring based on continuous feedback from beneficiaries via Community Offices Impact Evaluation  Rigorous design incorporating quantitative and qualitative methods, propensity score matching  Baseline, follow up surveys

Impact Evaluation Design Baseline carried out for all households in selected communities also as input to sub- project design. Baseline for sample households in control communities Follow up panel surveys in sample communities (project and control) one year after sub-project implementation, 5 years after, 9 years after. Over-sampling of households in follow up years to maintain sample sizes.

Data resources MIS: Designed under the project – includes data on spending, material inputs, number of beneficiaries, project status Household survey: Specially designed includes physical cadastre and socio-economic profile Focus Groups: In project communities Land and housing valuation: for sample households Epidemiological survey: Rapid survey already in use Secondary data: Existing administrative data for monitoring (e.g. spending, etc.), school records, health records, mortality.

Evaluation Indicators How has the project impacted on:  Basic infrastructure services (access, quality, use and affordability)  Housing improvements (land and housing values, tenure, ownership structure, private investment in housing, microfinance)  Economic outcomes (income and consumption, employment status, local economic activity)  Welfare outcomes (poverty and inequality)  Health outcomes (access to health care, infant mortality, malnutrition, water-borne disease, illness)  Education (school attendance, achievement, literacy)

Evaluation indicators, ctd. Security outcomes: (crime and violence) Migration (mobility in and out of slums) Environmental outcome (flooding, waste management, precarious housing) Participation and social capital (level of association and communal activity) Institutions (perceptions) Cost/benefit analysis Targeting analysis

Institutional arrangements CONDER responsible for management and implementation of M&E plan. (small unit) Baseline and follow up surveys contracted out to SEI, state-run statistical agency Assistance in design and analysis of impact evaluation to be contracted out, with input from the World Bank Supervision included in BB and with funds from DIME

Current Status Major delays in negotiations, implementation of project Major change in Government Baseline is now in the field

Some lessons The project will drive the evaluation, not the other way around ! Not appropriate for all projects, but there are some excellent opportunities to do IE. A cluster of thematic evaluations can really contribute to the knowledge base Best opportunities with interested TTL, Government counterpart Involve IE specialist from the start, continuity is very valuable Staffing can be a challenge, draw on local expertise if available Build supervision plans into the project Develop a prototype survey instrument Realistic expectations: measuring impacts requires time