impact evaluation for real-time decision-making Arianna Legovini Development Impact Evaluation (DIME)

Why are WE here today?

We are here to get better at what we do

Identify problems Test alternatives Adopt solutions A simple idea

research to inform a process of adaptive policy making How best do I get there? research to inform a process of adaptive policy making

Iterative learning Policy problems Data Hypotheses IE Testing Evidence/solutions Policy adoption Iterative learning Actionable answers

Rwanda Rural Roads: understanding the impact of road rehabilitation on national agricultural markets Set up high frequency national information and mapping on agricultural markets Measure effect of rural roads investments on: agricultural prices across national markets, availability of consumer goods and migration patterns

Objective to–over time—improve policy choices and policy effectiveness

2 types of inputs into decisions Data Evidence 2 types of inputs into decisions

Nairobi public transport IE: a system approach to road safety Set up real-time data system to measure movement of vehicles and people across the city Understand set of contractual arrangements between all actors in public transports Experiment with use of information and incentives to improve outcomes (mobility, road safety, emissions, etc)

What is a sufficient basis for a decision? Call for action “Lack of” sufficient basis for deciding to do something no data, no system of accountability, perverse incentives Diagnosis High quality data to understand problem and underlying population. Use it to debunk priors and formulate policy hypothesis. Test hypothesis and select policy modality Causal evidence to sort out complex supply & demand response to policy Use it to select most effective policy alternative.

The importance of being earnest Causal inference Scientifically valid tests to obtain actionable answers The importance of being earnest

what is causal inference?

the cause to get more of an effect Invest in the cause to get more of an effect Policy objective

enable good policy decisions Identify cause - effect enable good policy decisions Evaluation objective

CAUSALITY SO IMPORTANT? mmm…WHEN I USE MY UMBRELLA MY SHOES GET WET I THINK I WILL STOP USING UMBRELLAS WHY IS UNDERSTANDING CAUSALITY SO IMPORTANT? TO AVOID MISTAKES THAT CAN CAUSE HARM It is easy to confuse correlation with causation

Monitoring: trends and correlations not causality Monitoring tracks indicators over time (usually among participants) It is descriptive before-after analysis It tells us whether things are moving in the right direction It does not tell us why things happen (causality)

How do we sort our way through competing hypotheses?

YES NO NOT SURE Does the BRT or metro line increase incomes in the low-income areas we connect?

Invest in the metro Metro line announced Developers move in Increase in land and real estate values Outward migration of poorer population Metro line Increase access to jobs INCREASE INCOME

Dar Es Salaam BRT IE: evaluate & experiment to learn Evaluate BRT investments Impact of mass transit on improving labor market linkages Experiment with subsidies to understand cost of displacement and pricing strategies for access Dynamics of gentrification Strategies for inclusive urban growth

Ethiopia Highway and manufacturing growth Evaluate the corridor Unleasing new opportunities for growth in manufacturing Awasa biggest industrial park in Ethiopia Understand how to increase welfare Big labor market shock for young women in poor rural areas near industrial parks How do we protect them from exploitation?

Rio Supervia IE: social and economic aspects Measuring the incidence of harassment against women Estimating willingness to pay and economic costs of harassment

Evaluation Compares WHAT WOULD HAVE HAPPENED WHAT HAPPENED TO

What is counterfactual analysis? Compare same individual with & without intervention at the same point in time Missing data Compare statistically identical groups of individuals Comparable data

Counterfactual criteria Treated & control groups Have identical initial average characteristics (observed and unobserved) So that, the only difference is the treatment Therefore the only reason for the difference in outcomes is due to the treatment

Some important ideas Data must be embedded in a rigorous analytical framework to clarify: the process through which data is generated and how data can be interpreted. Strong team with research skills needed to: Impose analytical structure on design, implementation, measurement system and rigorous testing Critical for adaptive process to take place as to ensure greater effectiveness Skilled research team need the promise of analytical rewards

2 things will be fundamental The quality of the thinking and The quality of the engagement 2 things will be fundamental

Going from the impact of a road to thinking about the combinations of policies and solutions to maximize improvements in economic outcomes Quality of thinking

Each discovery leads to more refined and interesting questions It is by working together and thinking together that we can aim at solving problems and improving outcomes Quality of engagement

the idea generate data & evidence and motivate change Invest in impact evaluation research Capitalize on the World Bank/IFIs billion $ investments to pilot ideas and exert influence Empower policy people to exert control on their local environment generate data & evidence and motivate change

Develop institutional capacity for data & evidence-based policy Generate useful knowledge that solve development problems DIME objectives

We work through the policy cycle IE DESIGN IE IMPLEMENTATION IE DISSEMINATION Inform policy design Guide mid-course corrections Inform adoption and scale-up Train & apply Learn by doing Apply knowledge Systematic use of evidence Capacity building IE Products

Global outreach 5 regions 60 countries 173 IEs 300 partners

Across all sectors

USD 12 billion in underlying WB lending

What we learned doing this for the last 12 years It takes time to change the way projects operate It takes time to build the capacity, relationships, shared understanding It requires changing the skill composition of the teams Even when the skills are there, individuals may not have the bandwidth Need team members dedicated to the task It’s a journey not a destination

But we have also learned that Together, we can affect transformative change and have huge payoffs

Thank You! WEB http://dime.worldbank.org facebook.com/ieKnow #impacteval blogs.worldbank.org/impactevaluations microdata.worldbank.org/index.php/catalog/impact_evaluation