1. Development Impact Evaluation Initiative Innovations in investment climate reforms Paris, Nov 13, 2012 In collaboration with the Investment Climate Global Practice Causality and Evaluation Methods: Randomization Bilal Zia (DECFP) 1

2. Why Worry about Causality?  Imagine you are a naïve policymaker working in a country that suffers from persistent drought  On a lazy Sunday, you come across an article in the NY Times about the latest trend in fancy umbrellas  Funny, you note, how people tend to carry umbrellas in cities where it rains a lot

3. Why Worry about Causality? (2)  Empirically, this means that “carrying umbrellas” and “rain” are definitely correlated.

4. Why Worry about Causality? (2)  Empirically, this means that “carrying umbrellas” and “rain” are definitely correlated.  But…

5. Why Worry about Causality? (2)  Empirically, this means that “carrying umbrellas” and “rain” are definitely correlated.  But…  Does carrying an umbrella cause it to rain?

6. Correlations Can Lead to Bad Policy Advice  Suppose you conclude that umbrellas cause rain.  Policy advice: distribute free umbrellas in your drought ridden country!

7. Correlations Can Lead to Bad Policy Advice  Suppose you conclude that umbrellas cause rain.  Policy advice: distribute free umbrellas in your drought ridden country!  Not only does this make you look silly (and funny!), but it wastes valuable resources

8. A More Plausible Example Business Registration 1) Higher Profits

9. A More Plausible Example Rather than rubberstamping the program, questions should shift: -Are costs too high? - Are benefits too low? - Insufficient advertising? Business Registration OR 2) 1) Higher Profits Only highly profitable firms choose to register

10. Main Takeaway  Identifying causality matters for policy  This is not a joke academics play on others for fun  Funding, time, and other resources are at stake if we go down the wrong pathway, no matter how well intended it may be

11. How to Get to Causality?  Identify control/counterfactual group(s)!  Choose an evaluation methodology:  Randomization (prospective)  Observational Studies (backward looking)  Gather data

12. 12 Is this the impact of the program? Impact Assessment Example

13. 13 (+) Impact of the program (+) Impact of other (external) factors The Value of a Control Group

14. Impact Evaluation and Causality  Impact evaluation seeks to understand the causal effect of a program  separate the impact of the program from other factors  Need to find out what would have happened without the program, or with an alternative strategy 14

15. What is Impact Evaluation? Counterfactual analysis  Compare same firm with & without treatment, information etc. at the same point in time to measure the difference  This is impossible! The solution: Use a control group  Need to identify firms that represent what the treatment group would have been like if there was no project

16. Control Group Quality  But Control Group has to be good!  Projects started at specific times and places for particular reasons  What is a good control group?  By design treatment and comparison have the same characteristics (observed and unobserved), on average  Only difference is treatment  Control group represents what would have happened to the treatment population if the project had not occurred 16

17. Selection Bias  Can we just compare people who received the project to anyone who didn’t receive the project?  Danger of Selection Bias  What was the reason that some people received it and others didn’t?  Selection bias a major issue for impact evaluation  Projects started at specific times and places for particular reasons  Participants may select into programs (eligibility criteria)

18. How to create Control Group?  Need to find a group of non-treated people who can proxy for people who received treatment  This is hard: there is normally some reason why some people received treatment and others not, meaning any differences might not be due to the project

19. Randomized Experimental Design  Randomly assign potential beneficiaries to be in the treatment or comparison group  By design treatment and comparison have the same characteristics (observed and unobserved), on average  Only difference is treatment  With large sample, all characteristics average out  Unbiased impact estimates 19

20. Can we Randomize?  Randomization does not mean denying people the benefits of the project  Usually there are constraints within project implementation that allow randomization  Randomization is the fairest way to allocate treatment

21. Randomization is Very Flexible Must get the program whatever Not suitable for the program Randomize who gets the program

22. Unit of Randomization  For statistical power, randomizing at the individual level is best  Randomizing at higher level sometimes necessary:  Political constraints on differential treatment within community  Practical constraints—confusing for one person to implement different versions  Spillover effects may require higher level randomization  Randomizing at group level requires many groups because of within community correlation 22

23. Group or individual randomization?  Sample size and unit of randomization Individual randomizationGroup randomization N=16 N=4

24.  A good choice when there is incomplete take-up of the product of service  Those who get/receive promotion or marketing are more likely to enroll  But who got promotion or marketing was determined randomly, so not correlated with other observables or non-observables  Compare average outcomes of two groups: promoted/not promoted  Effect of offering the encouragement (Intent-To-Treat)  Effect of the intervention on the complier population (Local Average Treatment Effect) ▪ LATE= effect of offering program (ITT)/proportion of those who took it up

25. Opportunities to Randomize? Business Registration Example  Motivation: Businesses do not enter or formalize because of the associated time and financial costs (costs outweigh benefits)  Intervention: Reduce days to register from 10 – 5; reduce cost from $200 - $100  Expected result: increased firm entry and formalization, leading to increased competition and growth  But: What is the optimal registration fee to maximize participation and social benefits?

26. Test…a question…through variation in treatment Variation in intervention What is the price elasticity of demand for registration fees? Offer subsidies: $100; $50; $10 to different groups Complementary interventions Can financial training on banking products benefits from formalization? Offer training to a subset of businesses InformationWhat is the most effective way to communicate the new reform? Test phone calls, flyers, public events, radio adverts on different groups Firm behaviorDoes framing influence firm perceptions? Highlight the benefits of registering to one group and the threats/risks of not registering to another Supply side incentives Can staff recognition improve performance? Offer non-financial vs. financial incentives to processing staff.

27. Matching Grants Example  Motivation: Firms invest sub-optimally from a social perspective when public returns are greater than private. MGs can incentivize investment and increase social returns  Intervention: 50-50 matching grant to incentive investment in business dev. Services in the biotech industry  Expected result: Increased investment > improved competition and knowledge spillovers  But: What is the optimal matching amount?

28. Test…a question…through variation in treatment Variation in intervention What is the optimal matching proportion to maximize private investment? Vary matching amount: 25%; 50%; 75% Complementary interventions Can credit guarantees on the firm’s payment increase claim rate? Offer credit guarantees to a subset of firms InformationDo firms know best what support they need, or can they benefit from guidance? Give a group of potential applicants information on how various services may help their business and which service providers could offer help Firm behaviorHow do firms respond to deadlines? XXX Supply side incentives Can staff be incentivized to broaden the geographical reach of the program to support marginalized firms? Link performance payments to geographical targets for a subset of staff

29. What if Randomization Not Possible?  Randomization is the gold standard, though not the end all be all of evaluation  Many times, randomization is simply not possible or feasible:  Evaluate effect of exchange rate shift/inflation policy  Life saving vaccination (who wants to be in the control group?)  Interventions that have already happened  Other compelling scientific methods:  Difference in Difference / Fixed Effects  Regression Discontinuity Design  Instrumental Variables

30. Main Takeaway Source: http://www.xkcd.com