1. Copyright © 2015 Inter-American Development Bank. This work is licensed under a Creative Commons IGO 3.0 Attribution-Non Commercial-No Derivatives (CC-IGO BY-NC-ND 3.0 IGO) license (http://creativecommons.org/licenses/by-nc-nd/3.0/igo/legalcode) and may be reproduced with attribution to the IDB and for any non-commercial purpose. No derivative work is allowed.http://creativecommons.org/licenses/by-nc-nd/3.0/igo/legalcode Any dispute related to the use of the works of the IDB that cannot be settled amicably shall be submitted to arbitration pursuant to the UNCITRAL rules. The use of the IDB’s name for any purpose other than for attribution, and the use of IDB’s logo shall be subject to a separate written license agreement between the IDB and the user and is not authorized as part of this CC-IGO license. Note that link provided above includes additional terms and conditions of the license. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the Inter-American Development Bank, its Board of Directors, or the countries they represent.

2. Randomized Trials Francisco A. Gallego PUC-Chile and J-PAL LAC

3. Map Recapping Motivation Two-stage Randomized Trials Other Randomized Trials –With a selected sample –With partial compliance External Validity Additional Material: How to Randomize? 3

4. Map Recapping 4

5. Some key concepts: Impact evaluations in a context of causal inference Fundamental challenge in identifying a program’s impact –What would the life of an individual or a group be like without the program? NON-OBSERVABLE! –Estimating “non-observables”: control group or the counterfactual. Main problem: selection bias. –How? Observational or experimental methods 5

6. Map Recapping Motivation 6

7. In many fields, in the past, in medicine and today in the social sciences and public policy, the best evidence for counterfactuals is generated through randomized trials. Under certain conditions, randomized trials guarantee that the results of the control group really capture the counterfactual for a treatment group. 7

8. 8

9. Map Recapping Motivation Two-stage randomized trials 9

10. Randomization and Causal Inference Statisticians recommend a model with stages: –First stage: a random sample of units is selected from a defined population. –Second stage: this sample of units is randomly assigned to treatment and control groups. Randomizing in each of these stages has very different goals… 10

11. Basic Characterization of a Two-stage Randomized Trial 11 Target Population Not in the evaluation Randomized sample of the evaluation Random Assignment Treatment group Control group First Stage Second Stage

12. Randomization and Causal Inference First stage: seeks to guarantee external validity (but it is not a necessary or sufficient condition. More on this later) –Make sure that the sample used int he evaluation accurately represents the population (with a given sampling error) –Can be a stratified randomization Second stage: seeks to guarantee internal validity: –Make sure that the observed impact is due to the treatment and not to another factor (that is, it helps adequately define the counterfactual). –Can be a stratified randomization 12

13. Randomization and Causal Inference In this type of randomized trials (two-stages): If the sample is big (power calculations), it can be guaranteed that: Why? And therefore, Estimate ATE consistently. Why? 13

14. Map Recapping Motivation Two-stage randomized trials Other Randomized Trials –With a selected sample –With partial compliance 14

15. Randomization: Others In practice, however, there are other ways to randomize: –Typically, they deviate from the two-stage design, and because of this, it is interesting to understand what they can estimate and their limitations. We’ll see two cases: –Randomized trials without a first stage randomization Particular example: randomized trials with prior self-selection –Randomized trials with imperfect compliance 15

16. Map Recapping Motivation Two-stage randomized trials Other Randomized Trials –With a selected sample –Partial compliance Additional material: How to Randomize? 16

17. Randomized Evaluation with a Selected Sample 17 Target Population Not in the evaluation Selected sample of the evaluation Random Assignment Treatment Group Control Group

18. Randomized Evaluation with Volunteers 18 Target Population Not in the evaluation Volunteers Random Assignment Treatment Group Control Group

19. Randomized Evaluation With Volunteers It’s a self-selected sub-population (let V=1 if person applies) that applies to receive the treatment –Treated (D=1|V=1): those who wanted to participate and were assigned to the program with a lottery. –Control (D=0|V=1): those who wanted to participate and were not selected for the program. 19

20. Randomized Evaluation With Volunteers What does randomizing guarantee? And, as a result, our estimator consistently estimates for those who wanted to participate: That is, we are estimating the effect of the treatment on the treated and not on the entire population Is this a problem? When? According to Duflo et al. (2007), this is precisely the group that would be affected by the policy if it were scaled up, so it is also relevant to evaluate. 20

21. Map 21 Recapping Motivation Two-stage randomized trials Other Randomized Trials –With a selected sample –Partial compliance Additional material: How to Randomize?

22. Basic Characterization of a Randomized Evaluation with Imperfect Compliance 22 Target Population Not in the evaluation Evaluation sample Random Assignment Treatment Group Participants Non- participants Control Group Non- participants Treated controls

23. Partial Compliance with the Protocol Some individiuals in the control group receive the treatment. Ex. Deworming program (Miguel and Kremer, 2004); Baird et al. (2011): –Parents could try to change their children from the control school to the treatment school. –5% of students in the control group received the treatment. 23

24. Partial Compliance with the Protocol Some people in the treatment group decide not to take the treatment Ex. Deworming program: –Some students assigned to the treatment group in the treated schools did not receive the medical treatment. –78% of students assigned to the treatment group received some of the treatment. 24

25. Partial Compliance with the Protocol How do we estimate the effect of the program? Use the original assignment –If a girl ended up in a treatment school but was originally from the control group, she must be considered part of the control group when calculating the effect. This gives us the Intention to Treat (ITT) estimate 25

26. Intention to Treat (ITT) As we discussed before, ITT measures the average effect of offering the program. “What happened to the average child that is in one of the treated schools in this population?” Is this the right number to estimate? Is this the effect of deworming? 26

27. Think about concrete programs. In general, you can’t force individuals to ‘take’ a treatment. –For example, we could be interested not in the medical effect of deworming, but rather on what would happen with a real deworming program. –If students miss school frequently, and as a result don’t receive the deworming treatment, the ITT estimate could really be the most important one. When is ITT useful? 27

28. Partial Compliance with the Protocol 28 School 1Treatment? Received Treatment? Worm Load Student 1yes 0 Student 2yes 0 Student 3yes 1 Student 4yesno 3 Student 5yes 1 Student 6yesno 3 Student 7yesno 3 Student 8yes 1 Student 9yes 0 Student 10yesno 3 School 2Treatment? Received Treatment? Worm Load Student 1no 3 Student 2no 2 Student 3noyes 1 Student 4no 3 Student 5no 3 Student 6noyes 0 Student 7no 3 Student 8no 2 Student 9no 2 Student 10no 3

29. Some Possibilities 1.Compare those who RECEIVE the treatment with those who DO NOT receive the treatment 29 School 1Treatment? Received Treatment? Worm Load Student 1yes 0 Student 2yes 0 Student 3yes 1 Student 4yesno 3 Student 5yes 1 Student 6yesno 3 Student 7yesno 3 Student 8yes 1 Student 9yes 0 Student 10yesno 3 School 2Treatment? Received Treatment? Worm Load Student 1no 3 Student 2no 2 Student 3noyes 1 Student 4no 3 Student 5no 3 Student 6noyes 0 Student 7no 3 Student 8no 2 Student 9no 2 Student 10no 3

30. Some Possibilities 1.Compare those who RECEIVE the treatment with those who DO NOT receive the treatment -But, who receives the treatment is not random. -In the control group, those who receive the treatment may be the children of the most motivated parents. -In the treatment group, those who do not receive the treatment may be the children of the least motivated parents. 30

31. Some Possibilities 2.Compare those who RECEIVE the treatment in the treatment group with those who DO NOT receive the treatment in the control group 31 School 1Treatment? Received Treatment? Worm Load Student 1Yes 0 Student 2Yes 0 Student 3Yes 1 Student 4Yesno3 Student 5Yes 1 Student 6YesNo3 Student 7YesNo3 Student 8Yes 1 Student 9Yes 0 Student 10YesNo3 School 2Treatment? Received Treatment? Worm Load Student 1no 3 Student 2no 2 Student 3noyes1 Student 4no 3 Student 5no 3 Student 6noyes0 Student 7no 3 Student 8no 2 Student 9no 2 Student 10no 3

32. Some Possibilites 2.Compare those who RECEIVE the treatment in the treatment group with those who DO NOT receive the treatment in the control group -But, who receives the treatment is not random -In the treatment group, those who receive the treatment are the most motivated. -In the control group, those who do not receive the treatment are the least motivated. 32

33. Some Possibilities 3. Compare those who were ASSIGNED to the treatment group with those who were ASSIGNED to the control group 33 School 1Treatment? Received TreatmentWorm Load Student 1Yes 0 Student 2yes 0 Student 3yes 1 Student 4yesno3 Student 5yes 1 Student 6yesno3 Student 7yesno3 Student 8yes 1 Student 9yes 0 Student 10yesno3 School 2Treatment? Received Treatment?Worm Load Student 1no 3 Student 2no 2 Student 3noyes1 Student 4no 3 Student 5no 3 Student 6noyes0 Student 7no 3 Student 8no 2 Student 9no 2 Student 10no 3

34. Intention to Treat: What to do? 34 Average in school 1:(A) Average in school 2(B) Intention to School 1Treat?Treated? Student 1yes 4 Student 2yes 4 Student 3yes 4 Student 4yesno0 Student 5yes 4 Student 6yes no 2 Student 7 yes no0 Student 8yes 6 Student 9yes 6 Student 10yesno0 Avg change within school A= School 2 Student 1no 2 Student 2no 1 Student 3noyes3 Student 4no 0 Student 5no 0 Student 6noyes3 Student 7no 0 Student 8no 0 Student 9no 0 Student 10no 0 Avg change within school B = Observed changed In weight Intention to Treat Estimate 3 0,9 3,9 2,1

35. Partial Compliance with the Protocol ALWAYS respect the initial assignment when analyzing data, regardless of who ends up receiving the treatment. –The initial random assignment is the only random element and as such, useful for estimating impacts with internal validity. –Taking or not taking the treatment is not random Note that externalities from the treatment to the control can be thought of as a special case of 35

36. Partial Compliance with the Protocol Besides affecting our interpretation of the results, partial compliance has important implications on the design of the experiment. It affects power calculations and the sample size –It’s important to ANTICIPATE possible instances of partial compliance 36

37. Partial Compliance with the Protocol Can I estimate the impact of the treatment on those who effectively receive it? 37

38. Partial Compliance with the Protocol Sometimes… I need additional assumptions –Randomization is not enough anymore 38

39. Treatment on the Treated (TOT) Assumptions: 1.There are no treatment externalities –Children who don’t take deworming pills in school assigned to receive the treatment continue to get sick like before 2.There are no indirect effects from the treatment assignment –In randomized encouragement design, the encouragement does not have a direct impact on the results 39

40. Treatment on the Treated (TOT) The treatment effect on those who receive the treatment: –Assume that children who received the treatment had an increased weight, A, regardless of whethere they are in a treatment or control school –Assume that the children who did not receive the treatment had an increased weight of B. As in before, in both types of schools –We want to know A-B, the difference between the treated students and the students not treated 40

41. Treatment on the Treated (TOT) So... Y(T)=A*Prob[treated|T]+B(1-Prob[treated|T]) Y(C)=A*Prob[treated|C]+B(1-Prob[treated|C]) A-B= (Y(T)-Y(C)) / (Prob[treated|T] – Prob[treated|C]) = The “treatment on the treated” effect 41

42. Calculating Treatment on the Treated 42 Intention to School 1Treat?Treated? Student 1yes 4 Student 2yes 4 Student 3yes 4 A =Increases if treated Student 4yesno0 B =Increases if not treated Student 5yes 4 Student 6yesno2 Student 7yesno0 ToT estimator: A-B Student 8yes 6 Student 9yes 6 Student 10yesno0 A-B = Y(T)-Y(C) Average change Y(T)= Prob(Treated|T)-Prob(Treated|C) School 2 Student 1no 2 Y(T) Student 2no 1 Y(C) Student 3noyes3 Prob(Treated|T) Student 4no 0 Prob(Treated|C) Student 5no 0 Student 6noyes3 Student 7no 0 Y(T)-Y(C) Student 8no 0 Prob(Treated|T)-Prob(Treated|C) Student 9no 0 Student 10no 0 Average change Y(C) = A-B Observed Change in weight 3 0,9 3 60% 20% 2,1 40% 5,25 Compare the Intention to Treat: 2,1

43. Treatment on the Treated (TOT) In practice, TOT is estimated using the treatment assignment as an instrumental variable for receiving the treatment. –We’ll see more of this later… –Consequently, the estimated effect is a local effect. 43

44. Map Recapping Motivation Two-stage randomized trials Other randomized trials –With a selected sample –With partial compliance External Validity Additional Material: How to Randomize? 44

45. External Validity Internal validity is a necessary condition for the results of a randomized trial to be generalizable… But it is not enough… 45

46. Threats to External Validity: Behavioral Responses to Evaluations A limitation of randomized trials is that the evaluation by itself can provoke a change in the behavior of individuals in the treatment and control groups. –When the behavior of the treatment group changes: Hawthorne Effect. –When the behavior of the comparison group changes: John Henry Effect. 46

47. Behavioral Responses to Evaluations Furthermore, a program can generate behavioral responses that wouldn’t occur if the program were generalized. 47

48. Generalizability of Results Depends on three factors: –Program implementation: can it be replicated at a large scale? Nationally? Program implementation General equilibrium effects –Sample: is it representative? –Sensibility to results: would a similar program, but slightly different, have the same effect? 48

49. Map Recapping Motivation Two-stage randomized trials Other randomized trials –With a selected sample –With partial compliance External Validity Additional Material: How to Randomize? 49

50. Unit and method of randomization Real world restrictions Reviewing the unit and the method Variations in simple treatment-control analysis How to Randomize? 50

51. Understand how to randomize (mechanics) and decide between alternatives –Key concepts: unit and method of randomization, stratification. How to (creatively) adapt randomization to the restrictions imposed by the real world Understand some variations in the treatment-control analysis Objectives 51

52. Cómo aleatorizar, Parte I - 52 Random Assignment 2006 Monthly per-capita income, in rupees. 1000 500 0 T C 14571442

53. Randomization Mechanics It’s necessary to have a sampling frame (list from which to randomize) Options: –Names out of a hat –Random number generator in a spreadsheet and organize observations randomly. –Use a code from STATA What if there’s no prior list? 53 Source: Chris Blattman

54. Unit of Randomization 1.Randomize at the individual level 2.Randomize at the group level “Randomized trial with clustered observations” At what level should we randomize? 54

55. Unit of Randomization: Some Considerations What unit does the program’s treatment target? What is the unit of analysis? 55

56. Randomization Unit: Individual? 56

57. Randomization Unit: Individual? 57

58. “Groups of people”: Randomized trial with grouped units Randomization Unit: Grouped Units?

59. Randomization Unit: Class? 59

60. Randomization Unit: Class? 60

61. Randomization Unit: School? 61

62. Randomization Unit: School? 62

63. Randomization Unit: Target Population How is the intervention administered? What is the catchment area of each “intervention unit”? What is the scope of the possible impact? 63

64. Randomization Unit: Analysis Remember: What is our unit of measurement for impact? How/where are we going to find that data? 64

65. Lottery: Begin With Clinical Trials as a Reference Point Take 1000 people and give a medicine to half of them Can we apply this approach to social programs? 65

66. Randomly chosen from a pool of applicants. Participants know “winners” and “losers” Simple lottery is useful when there is no a priori reason to discriminate Perceived as fair Transparent Tends to be politically viable Lotteries are simple, common, and transparent 66

67. Restrictions: Resources Why are resource restrictions the best friend of an evaluator? Many programs have limited resources There are more eligible people than resources available to serve them Very common in practice: –Training for entrepreneurs or farmers –Vouchers in education –Cash transfers 67

68. Restrictions: Contamination Remember the counterfactual If the control group is different to the counterfactual, our results may be biased 68

69. Restriction: Contamination Externality/Treated Control Externality Treated control Partial compliance or non-compliance with the treatment 69

70. Restrictions: Logistics Assume that administering the deworming pills is part of the work of a health worker. Assume that the health worker already gave services to people in the control and treatment group. It could be difficult to train them to follow different procedures for those in each group and keep a registry of what is administered to whom. 70

71. Restrictions: Justice, Politics Randomize at the child level within classrooms Randomize at the classroom level within schools Randomize at the community level 71

72. Restrictions: Sample Size The program is only big enough to benefit a handful of communities. 72

73. What if you have 500 applicants for 500 slots? Consider uncommon lottery designs You could advertise the program more… Is this ethical? 73

74. Sometimes, the filter matters Assume that there are 2000 applicants The filter produces 500 eligible candidates There are 500 slots A simple lottery won’t work. What are our options? 74

75. Consider filtering rules or targeting What are you filtering for? What elements are essential? Selection procedures can only exist to reduce the number of eligible candidates in order to meet a capacity restriction or to target the program. If certain filtering mechanism appear as arbitrary (but not random) the randomization can be useful as a filter and help us evaluate 75

76. Randomization in the Bubble Sometimes, a partner may not be willing to randomize among eligible people. A partner may be willing to randomize “in the bubble” People “in the bubble” are people who are on the margins of eligibility. Just above a threshold  not eligible, but almost. What effect are we measuring? What does it mean for external validity? 76

77. Randomization “in the bubble” Within the bubble, compare the treatment with the control Participants Non-participants Treatment Control 77

78. When Analysis Matters: Partial Lottery Program administrators can still have discretion Example: training program Example: expansion of a consumption credit in South Africa 78

79. Phase-In Design: Takes Advantage of Scaling Eventually, everyone receives the treatment Naturally, when scaling up there are resource restrictions What determines that schools, etc., will be included each year? 79

80. Phase-in Design Round 1 Treatment: 1/3 Control: 2/3 Round 2 Treatment: 2/3 Control: 1/3 Round 3 Treatment: 3/3 Control: 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 Round 1 Treatment: 1/3 Control: 2/3 Round 2 Treatment: 2/3 Control: 1/3 Randomized Trial Ends 80

81. Phase-in Design Advantages Eventually, everyone receives something Provides incentives to maintain contact Concerns It can complicate the estimate of effect sizes through time Attention to phase-in windows Do expectations change behavior today? 81

82. Rotation Design Groups receive the treatment in turns Advantages Concerns 82

83. Round 1 Treatment: 1/2 Control: 1/2 Rotation Design Round 2 Treatment from Round 1  Control —————————————————————————— Control from Round 1  Treatment Round 1 Treatment: 1/2 Control: 1/2 83

84. “You want to survey me? Then treat me!” Phased-in design may not provide sufficient benefits to participants from the latter rounds Cooperation of the control group may be essential Consider this within-group randomization For example, balsakhi program (Banerjee et al. 2007) All participants obtain some benefit. Concern: greater chance of contamination 84

85. Encouragement Design: What to do when you can’t randomize access? Sometimes, it’s practically or ethically impossible to randomize access to the program However, most programs don’t have a 100% take up rate Randomize encouragements to receive treatment 85

86. Encouragement Design Encouragement No Encouragement Participated Didn’t participate Compliance Non-compliance compare encouraged with not encouraged Don’t compare participants with non-participants Adjust for non-compliance when analyzing. These have to be correlated 86

87. What is an encouragement? Something that makes some people more likely than others to use a program. It’s not a “treatment” by itself For whom are we calculating the treatment effect? Think about who responds to the encouragement 87

88. To Recap: Possible Designs Simple Lottery Randomization “in the bubble” Phased-in randomization Rotation Encouragement Design Note that these are not mutually exclusive 88

89. Randomization Methods: Recapping 89 Design Most Useful When AdvantagesDisadvantages Simple Lottery Oversubscribed programFamiliarControl group may not cooperate It’s ok if some people don’t receive anything Easy to Understand Differential attrition Easy to implement Can be implemented in public Phase In Expands with timeEasy to understand Anticipating treatment can change Short term behavior Finally, everyone has to receive The treatment Restrictions are easy to explain Difficult to measure long-term effects Control group accepts given that it expects Future benefits Rotation Everyone has to receive the treatment at Some point but there are not enough Resources for everyone at the same time More data than in phase-in designDifficult to measure long-term effects Encouragement Program has to be open to all Can randomize at the individual level even When the program is given at the Individual level Measures the impact on those who Respond to the incentive When the take up rate is low, but you can Use incentives to encourage people to participate Needs an incentive sufficiently big to Encourage participation. The encouragement can have a Direct effect