1. impact evaluation for real-time decision-making
Arianna Legovini
Development Impact Evaluation (DIME)

2. Why are WE here today?

3. We are here to get better at what we do

4. Identify problems
Test alternatives
Adopt solutions
A simple idea

5. research to inform a process of adaptive policy making
How best do I get there?
research to inform a process of adaptive policy making

6. Iterative learning Policy problems Data Hypotheses IE Testing
Evidence/solutions
Policy adoption
Iterative learning
Actionable answers

7. Rwanda Rural Roads: understanding the impact of road rehabilitation on national agricultural markets
Set up high frequency national information and mapping on agricultural markets
Measure effect of rural roads investments on:
agricultural prices across national markets,
availability of consumer goods and
migration patterns

8. Objective to–over time—improve policy choices and policy effectiveness

9. 2 types of inputs into decisions
Data
Evidence
2 types of inputs into decisions

12. Nairobi public transport IE: a system approach to road safety
Set up real-time data system to measure movement of vehicles and people across the city
Understand set of contractual arrangements between all actors in public transports
Experiment with use of information and incentives to improve outcomes (mobility, road safety, emissions, etc)

13. What is a sufficient basis for a decision?
Call for action
“Lack of” sufficient basis for deciding to do something
no data, no system of accountability, perverse incentives
Diagnosis
High quality data to understand problem and underlying population.
Use it to debunk priors and formulate policy hypothesis.
Test hypothesis and select policy modality
Causal evidence to sort out complex supply & demand response to policy
Use it to select most effective policy alternative.

14. The importance of being earnest
Causal inference
Scientifically valid tests to obtain actionable answers
The importance of being earnest

15. what is causal inference?

16. the cause to get more of an effect
Invest in
the cause to get more of an effect
Policy objective

17. enable good policy decisions
Identify
cause - effect
enable good policy decisions
Evaluation objective

18. CAUSALITY SO IMPORTANT?
mmm…WHEN I USE MY UMBRELLA MY SHOES GET WET
I THINK I WILL STOP USING UMBRELLAS
WHY IS UNDERSTANDING
CAUSALITY SO IMPORTANT?
TO AVOID MISTAKES THAT CAN CAUSE HARM
It is easy to confuse correlation with causation

19. Monitoring: trends and correlations not causality
Monitoring tracks indicators over time
(usually among participants)
It is descriptive before-after analysis
It tells us whether things are moving in the right direction
It does not tell us why things happen (causality)

20. How do we sort our way through competing hypotheses?

21. YES NO
NOT SURE
Does the BRT or metro line increase incomes in the low-income areas we connect?

22. Invest in the metro Metro line announced Developers move in
Increase in land and real estate values
Outward migration of poorer population
Metro line
Increase access to jobs
INCREASE INCOME

23. Dar Es Salaam BRT IE: evaluate & experiment to learn
Evaluate BRT investments
Impact of mass transit on improving labor market linkages
Experiment with subsidies to understand cost of displacement and pricing strategies for access
Dynamics of gentrification
Strategies for inclusive urban growth

24. Ethiopia Highway and manufacturing growth
Evaluate the corridor
Unleasing new opportunities for growth in manufacturing
Awasa biggest industrial park in Ethiopia
Understand how to increase welfare
Big labor market shock for young women in
poor rural areas near industrial parks
How do we protect them from exploitation?

25. Rio Supervia IE: social and economic aspects
Measuring the incidence of harassment against women
Estimating willingness to pay and economic costs of harassment

26. Evaluation
Compares
WHAT WOULD HAVE HAPPENED
WHAT HAPPENED TO

27. What is counterfactual analysis?
Compare same individual
with & without intervention
at the same point in time
Missing data
Compare statistically identical groups of individuals
Comparable data

28. Counterfactual criteria
Treated & control groups
Have identical initial average characteristics (observed and unobserved)
So that, the only difference is the treatment
Therefore the only reason for the difference in outcomes is due to the treatment

29. Some important ideas
Data must be embedded in a rigorous analytical framework to clarify:
the process through which data is generated and
how data can be interpreted.
Strong team with research skills needed to:
Impose analytical structure on design, implementation, measurement system and rigorous testing
Critical for adaptive process to take place as to ensure greater effectiveness
Skilled research team need the promise of analytical rewards

30. 2 things will be fundamental
The quality of the thinking and
The quality of the engagement
2 things will be fundamental

31. Going from the impact of a road to thinking about the combinations of policies and solutions to maximize improvements in economic outcomes
Quality of thinking

32. Each discovery leads to more refined and interesting questions
It is by working together and thinking together that we can aim at solving problems and improving outcomes
Quality of engagement

33. the idea generate data & evidence and motivate change
Invest in impact evaluation research
Capitalize on the World Bank/IFIs billion $ investments to pilot ideas and exert influence
Empower policy people to exert control on their local environment
generate data & evidence and motivate change

34. Develop institutional capacity for data & evidence-based policy
Generate useful knowledge that solve development problems
DIME objectives

35. We work through the policy cycle
IE DESIGN
IE IMPLEMENTATION
IE DISSEMINATION
Inform policy design
Guide mid-course corrections
Inform adoption and scale-up
Train & apply
Learn by doing
Apply knowledge
Systematic use of evidence
Capacity building
IE Products

36. Global outreach
5 regions
60 countries
173 IEs
300 partners

37. Across all sectors

38. USD 12 billion in underlying WB lending

39. What we learned doing this for the last 12 years
It takes time to change the way projects operate
It takes time to build the capacity, relationships, shared understanding
It requires changing the skill composition of the teams
Even when the skills are there, individuals may not have the bandwidth
Need team members dedicated to the task
It’s a journey not a destination

40. But we have also learned that
Together, we can affect transformative change and have huge payoffs

41. Thank You! WEB http://dime.worldbank.org facebook.com/ieKnow
#impacteval
blogs.worldbank.org/impactevaluations
microdata.worldbank.org/index.php/catalog/impact_evaluation