1. How to randomize? povertyactionlab.org

2. Quick Review: Why Randomize Choosing the Sample Frame Choosing the Unit of Randomization Options How to Choose Randomization Methods Mechanics of randomization Stratification Variations on Simple Treatment-Control

3. Ensure that we compare oranges and oranges Randomization: randomly select treatment and comparison group before program starts If we construct a comparison group after the program has started  Bias – Before and After Other things may be going on – Participants and Non participants (Simple comparison; Diff-in-Diff; Regression) Selection Bias

4. Difference of Means Intrinsic DifferencesImpact of the program ObservablesNon Observables Selection Bias

5. Baseline Randomization Assignment to Treatment Assignment to Control Sample frame Compare T and C

6. How to Randomize, Part I - 5 Random assignment 2006 Income per person, per month, rupees 1000 500 0 Treat Compare 14571442

7. Baseline Randomization Assignment to Treatment Assignment to Control Endline Compare T and C Sample frame Intervention Compare T and C

8. Quick Review: Why Randomize Choosing the Sample Frame Choosing the Unit of Randomization Options How to Choose Randomization Methods Mechanics of randomization Stratification Variations on Simple Treatment-Control

9. Geographic area and Population Where will the project occur? – Example: Purnea, Ajmer Districts Who is the target population? What types of tolas/villages will we be working in? Choice of area and population is usually not random – Convenience, eligible population How representative do we want to be?

10. Quick Review: Why Randomize Choosing the Sample Frame Choosing the Unit of Randomization Options How to Choose Randomization Methods Mechanics of randomization Stratification Variations on Simple Treatment-Control

11. 1.Randomizing at the individual level 2.Randomizing at the group level “Cluster Randomized Trial” 10

14. “Groups of individuals”: Cluster Randomized Trial Unit of Randomization: Clusters?

19. Quick Review: Why Randomize Choosing the Sample Frame Choosing the Unit of Randomization Options How to Choose Randomization Methods Mechanics of randomization Stratification Variations on Simple Treatment-Control

20. Spillovers and Crossovers – How wide is the potential impact? How is the intervention administered? – What is the mobilization method? – What is the catchment area of each “unit of intervention”? Others – Logistical constraints – Budget and Sample size requirements

21. Remember the counterfactual! If control group is different from the counterfactual, results will be biased Can occur due to Spillovers Crossovers

22. TreatmentControl Spillovers Counterfactual

23. Positive spillovers (treatment -> control) – Medicines or vaccines – District gives more resources to control schools Negative spillovers (control -> treatment) – Re-infection Negative spillovers (treatment -> control) – Less diseases in the area so parents put les efforts in treating kids Mother Literacy? How do we ensure this does not happen?

24. Control individuals get directly treated – E.g Balsakhis: Control kids get treated (e.g. transfer to treatment schools) – E.g: Microfinance evaluation in urban areas Mother Literacy? How do we ensure this does not happen?

25. What is the mobilization method? What is the catchment area of each “unit of intervention”? Usually best to keep the randomization unit as the unit of intervention Given this, also useful to keep the unit of randomization as small as possible

26. Logistics – Large villages as unit of randomization: more travel, difficult to find volunteers etc. Budget and sample size – Individual randomization is cheaper (tomorrow!) – There may not be enough clusters of a certain type available for the intervention Villages vs tolas

27. Quick Review: Why Randomize Choosing the Sample Frame Choosing the Unit of Randomization Options How to Choose Randomization Methods Mechanics of randomization Stratification Variations on Simple Treatment-Control

28. Most programs have limited resources – Vouchers, Farmer Training Programs – Results in more eligible recipients than resources will allow services for Programs are often piloted in a smaller scale before being scaled up Lottery can be a fair way to select participants

29. What if you have just enough resources for the number of recipients in your work area? – 500 seats for a training, and 500 applicants Increase outreach of activities May not be possible Control group may be upset anyway Phase-in Design

30. Everyone gets program eventually Natural approach when expanding program faces resource constraints What determines which schools, branches, etc. will be covered in which year? – Some choices based on need, geography, etc. – Others largely arbitrary

31. Phase-in design Round 1 Treatment: 1/3 Control: 2/3 Round 2 Treatment: 2/3 Control: 1/3 Round 3 Treatment: 3/3 Control: 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 Round 1 Treatment: 1/3 Control: 2/3 Round 2 Treatment: 2/3 Control: 1/3 Randomized evaluation ends

32. Advantages Everyone gets something eventually Concerns Can complicate estimating long-run effects Natural expansion timeline may be too short Do expectations of change changes behavior today?

33. Groups get treatment in turns Advantages Concerns How to Randomize, Part I - 32

34. Round 1 Treatment: 1/2 Control: 1/2 Rotation design Round 2 Treatment from Round 1  Control —————————————————————————— Control from Round 1  Treatment Round 1 Treatment: 1/2 Control: 1/2

35. Phase-in may not provide enough benefit to late round participants Cooperation from control group may be critical Consider within-group randomization E.g., balsakhi program All participants get some benefit Concern: increased likelihood of contamination

36. Sometimes it’s practically or ethically impossible to randomize program access But most programs have less than 100% take- up Randomize encouragement to receive treatment Encouragement: Something that makes some folks more likely to use program than others

37. Encouragement design Encourage Do not encourage participated did not participate Complying Not complying compare encouraged to not encouraged do not compare participants to non-participants adjust for non-compliance in analysis phase These must be correlated

38. MethodMost useful whenAdvantagesDisadvantages Lottery Limited resources Piloting a new program Easy to understand and implement Control group may be upset Phase in Program expanding over time Everyone must get treatment Easy to understand and explain Control group not upset Anticipation of treatment Long term effects RotationEveryone must get something No control group upset Spillovers more likely More difficult to measure long term Encouragement Program has to be done everywhere Take up low but can easily be increased Can easily randomize at individual level Need very effective incentive Impact on those who respond to incentive

39. Quick Review: Why Randomize Choosing the Sample Frame Choosing the Unit of Randomization Options How to Choose Randomization Methods Mechanics of randomization Stratification Variations on Simple Treatment-Control

40. Need sample frame Pull out of a hat/bucket 39 Source: Chris Blattman

41. Need sample frame Pull out of a hat/bucket Use random number generator in spreadsheet program to order observations randomly Stata program code 40 Source: Chris Blattman

43. Need sample frame Pull out of a hat/bucket Use random number generator in spreadsheet program to order observations randomly Stata program code What if no existing list? 42 Source: Chris Blattman

44. Quick Review: Why Randomize Choosing the Sample Frame Choosing the Unit of Randomization Options How to Choose Randomization Methods Mechanics of randomization Stratification Variations on Simple Treatment-Control

45. Objective: balancing your sample Ultimately increases your power What is it: – dividing the sample into different subgroups – selecting treatment and control from each subgroup What happens if you don’t stratify? 44

46. VillageTypeStatus uyufssgLow litT AdjkjdkLow litC afiudiufaHigh LitT aduyuLow litC afjhHigh litT askfjkfjkLow litC fdkvofjHigh litT daisudimvLow litC vadlsvkoiLow litT VaofdivofdaiLow litC caidsudiauHigh LitT

47. VillageTypeStatus uyufssgLow litT AdjkjdkLow litC afiudiufaLow litT aduyuLow litC afjhLow litT askfjkfjkLow litC fdkvofjLow litT daisudimvHigh LitT vadlsvkoiHigh LitC VaofdivofdaiHigh LitT caidsudiauHigh LitC

48. Stratify on variables that could have important impact on outcome variable (bit of a guess) Logistics Stratify on subgroups that you are particularly interested in (where may think impact of program may be different) Stratification more important when small data set Can get complex to stratify on too many variables Makes the draw less transparent the more you stratify 47

49. Quick Review: Why Randomize Choosing the Sample Frame Choosing the Unit of Randomization Options How to Choose Randomization Methods Mechanics of randomization Stratification Variations on Simple Treatment-Control

50. Sometimes core question is deciding among different possible interventions You can randomize these programs Test different components of treatment in different combinations Test whether components serve as substitutes or compliments What is most cost-effective combination? How to Randomize, Part I - 49

51. Treatment 1 Treatment 2 Treatment 3 Multiple treatments

52. Randomization Intervention 1: Mothers Literacy Classes 60 villages Intervention 2: Train Mothers to Interact with Children’s Education 60 Villages Intervention 3: Combine Intervention 1 and 2 60 Villages Comparison group 60 Villages 240 Elligible Villages In Each State:

53. Some schools are assigned full treatment – All kids get pills Some schools are assigned partial treatment – 50% are designated to get pills Testing subsidies and prices E.g: Information campaign - how much do you need to spread information?

54. Combined 155 clusters Treatment schools Control schools Random Assignment Treatment individuals Control individuals