1. Development Impact Evaluation in Finance and Private Sector

2. Arianna Legovini
Head, Development Impact Evaluation Initiative (DIME)
World Bank
Impact Evaluation for Real Time Decision-Making in Finance and Private Sector

3. Do we know…
How much do regulatory reforms and one-stop-shops generate entry and employment? What incentives can be used to convince local politician to implement the reforms?
What information and services improve business organization and lower production costs?
What networking and market access structures lower firm death rates? What type of trade facilitation will increase exports and export diversification?

4. Trial and error
We turn to our best judgment for guidance and pick a package of services, access to credit mechanisms, registration rules, trade facilitation incentives …
Is there any other package, mechanism, rule or incentive that will do better?

5. The decision process is complex
Design
Early roll out
Implementation
A few big decisions are taken during design but many more decisions are
taken during roll
out & implementation

6. Developing a decision tree for an SME support program…
SME services
Public information campaign
Non-financial services
Center-based training
Vouchers + information
On-site training
Non-financial + financial services
Door-to-door promotion

7. How to select between plausible alternatives?
Establish which decisions will be taken upfront and which will be tested during roll-out
Experimentally test critical nodes: measure the impact of one option relative to another or to no intervention
Pick better and discard worse during implementation
Cannot learn everything at once
Select carefully what you want to test by involving all relevant partners

8. Walk along the decision tree for your SME program to get more results
SME services
Public information campaign
Non-financial services
Center-based training
Vouchers + information
On-site training
Non-financial + financial services
Door-to-door promotion
Take-up
MAX
Use of services
Benefits from services

9. H0w we support you

10. What is Impact Evaluation?
Impact evaluation measures the effect of an intervention on outcomes of interest relative to a counterfactual (what would have happened in the absence of)
It identifies the causal effect of an intervention on an outcome separately from the effect of other time-varying conditions

11. What is counterfactual analysis?
Compare same individual with & without training, information etc. at the same point in time to measure the effect
This is impossible
Impact evaluation uses large numbers of entrepreneurs, or municipalities to estimate the effect

12. What is a good counterfactual?
Treated & control groups have identical observed and unobserved characteristics
The only reason for the difference in outcomes is due to the intervention
How?
Assign intervention to some and not some other eligible populations on a random basis or on the basis of clear and measurable criteria
Obtain a treatment and a control group
Measure and compare outcomes in those groups over time

13. Ethical considerations
It is not ethical to deny benefits to something that is available and we know works
HIV medicine proven to prolong life
It is ethical to test interventions before scale up if we don’t know if it works and whether it has unforeseen consequences
Food aid may impair local markets and create perverse incentives
Most times we use opportunities created by roll out and budget constraints to evaluate so as to minimize ethical considerations

14. Methods (later & tomorrow)
Experimental or random assignment
Equal chance of being in the treatment or comparison group
By design treatment and comparison have the same characteristics (observed and unobserved), on average
Simple analysis (means comparison) and unbiased impact estimates
Non-experimental (Regression discontinuity, IV and encouragement designs, Difference in difference)
Require more assumptions or might only estimate local treatment effects
May suffer from non-observed variable bias
Use more than one method to check robustness of results

15. How is monitoring different from impact evaluation?
Monitoring is trend analysis
Change over time
Compare results before and after on the “treated” group
After
Before A B t0 t1
Intervention
Change Y B’
Impact
Impact evaluation
Change over time and relative to comparison
Compare results before and after in the “treated” group and relative to the “untreated” group

16. Monitoring & Impact Evaluation
monitoring to track implementation efficiency
(input-output)
impact evaluation to measure effectiveness (output-outcome)
BEHAVIOR
MONITOR EFFICIENCY
INPUTS
OUTPUTS
OUTCOMES
EVALUATE EFFECTIVENESS
$$$

17. Question types & methods
Monitoring and process evaluation
Is program being implemented efficiently?
Is program targeting the right population?
Are outcomes moving in the right direction?
Impact Evaluation
What was the effect of the program on outcomes?
How would outcomes change under alternative program designs?
Is the program cost-effective?
Descriptive
analysis
Causal
analysis

18. Why Evaluate? Improve quality of programs
Separate institutional performance from quality of intervention
Test alternatives and inform design in real time
Increase program effectiveness
Answer the “so what” questions
Build government institutions for evidence-based policy-making
Plan for implementation of options not solutions
Find out what alternatives work best
Adopt better way of doing business and taking decisions

19. Institutional framework
PM/Presidency:
Communicate to constituencies
Treasury/
Finance:
Allocate budget
Line ministries:
Deliver programs and negotiate budget
Cost-effectiveness of different programs
Effects of government program
BUDGET
SERVICE
DELIVERY
CAMPAIGN
PROMISES
Accountability
Cost-effectiveness of alternatives and effect of sector programs

20. Shifting Program Paradigm
From:
Program is a set of activities designed to deliver expected results
Program will either deliver or not
To:
Program is menu of alternatives with a learning strategy to find out which work best
Change programs overtime to deliver more results

21. Shifting Evaluation Paradigm
From retrospective, external, independent evaluation
Top down
Determine whether program worked or not
To prospective, internal, and operationally driven impact evaluation /externally validated
Set program learning agenda bottom up
Consider plausible implementation alternatives
Test scientifically and adopt best
Just-in-time advice to improve effectiveness
of program over time

22. Retrospective (designed & evaluated ex-post) vs
Retrospective (designed & evaluated ex-post) vs. Prospective (designed ex-ante and evaluated ex-post)
Retrospective impact evaluation:
Collecting data after the event you don’t know how participants and nonparticipants compared before the program started
Have to try and disentangle why the project was implemented where and when it was, after the event
Prospective evaluation:
design the evaluation to answer the question you need to answer
collect the data you will need 22

23. Is this a one shot analytical product?
Must provide useful (actionable) information at each step of the impact evaluation

24. Thank you
Financial support from
Is gratefully acknowledged