Class 7 – Randomization* *(how to design an RCT that accommodates real world constraints)

Randomly sample from area of interest

Randomly sample from area of interest Randomly assign to treatment and control Randomly sample From both treatment and control

BEFORE YOU EVEN TALK ABOUT RANDOMIZATION DESIGN…  Explain why randomization is necessary  Talk about attribution  Articulate why (non-random) comparison group may be different and specify way this may bias results  Explain that the intervention may have unintended (good or bad) consequences worth measuring  Can measure spillover  Cost-benefit analyses

 Unit of Randomization  Ways to Randomize  Simple Randomization (eg: lottery)  Randomization “in the bubble”  Phase-in  Rotation  Encouragement Design  Multiple treatments, 2 stage  (if time) Stratification

 Options  Individual  Cluster  Which level to randomize?  Considerations  What unit does the program target for treatment?  What is the unit of analysis?

“Groups of individuals”: Cluster Randomized Trial

 In the example discussed last week, students were tutored after school   how did we randomize last week?  What if students are taken out of class to be tutored, say by a teaching assistant?   how would you randomize?

 Randomizing at the child-level within classes  Randomizing at the class-level within schools  Randomizing at the community-level

 Nature of the Treatment  How is the intervention administered?  What is the catchment area of each “unit of intervention”  How wide is the potential impact?  Aggregation level of available data  Do you want to measure spillover?  Power requirements  Generally, best to randomize at the level at which the treatment is administered.

 Sometimes a program is only large enough to serve a handful of communities  Primarily an issue of statistical power  Will be addressed next week

 Unit of Randomization  Ways to Randomize  Simple Randomization (eg: lottery)  Randomization “in the bubble”  Phase-in  Rotation  Encouragement Design  Multiple treatments, 2 stage  (if time) Stratification

“high rate” “low rate” Final sample included 132 branch offices in 80 geographic clusters

 School Voucher Program in Colombia (PACES)  Student must be entering 6th grade and under 15 years old  Students must provide evidence that they live in poor neighborhood  Renewable through graduation unless student is retained in a grade  Vouchers awarded by lottery if demand exceeds supply  Covered about 60% of fees  Key Findings (after 3 years) on Voucher Recipients:  Increased Usage of Private Schools  Higher Educational Attainment  No Difference in Drop-out Rates  Less Grade Repetition  Higher Test Scores  Less Incidence of Teen-age Employment  Further research on peer effects

 Lotteries are simple, common and transparent  Randomly chosen from applicant pool  Participants know the “winners” and “losers”  Simple lottery is useful when there is no a priori reason to discriminate  Perceived as fair  Transparent

 What if you have 500 applicants for 500 slots?  Could increase outreach activities  (but think of external validity)  Sometimes screening matters  Suppose there are 2000 applicants for 500 slots  Screening of applications produces 500 “worthy” candidates  A simple lottery will not work - What are our options?

What are they screening for? Which elements are essential? Selection procedures may exist only to reduce eligible candidates in order to meet a capacity constraint If certain filtering mechanisms appear “arbitrary” (although not random), randomization can serve the purpose of filtering and help us evaluate

 Organizations may not be willing to randomize among eligible people.  But might be willing to randomize those at the margin – ie, those who are borderline in terms of eligibility  Just above the threshold  not eligible, but almost  What treatment effect do we measure? What does it mean for external validity?  (hint: review RDD from last week)

 Dean Karlan and Jonathan Zinman worked with a bank in South Africa  Loan officers ranked applicants as “egregiously uncreditworthy” or “marginally uncreditworthy”  Randomly selected marginal applicants to be reconsidered  53% of those were offered a loan  Look at impact of credit on those randomized into the “reconsider group”  Not just those offered loan… more on this next week.  Found that access to credit increased likelihood that clients would retain job, increase income, feel less food insecurity  Marginal loans are profitable, but less than regular loans

 Take advantage of operational constraints  Typical during expansion phase  Everyone gets program eventually  Figure out what determines order of expansion  Examples: Progresa (Mexico), Deworming (Kenya)

Phase-in design Round 1 Treatment: 1/3 Control: 2/3 Round 2 Treatment: 2/3 Control: 1/3 Round 3 Treatment: 3/3 Control: Round 1 Treatment: 1/3 Control: 2/3 Round 2 Treatment: 2/3 Control: 1/3 Randomized evaluation ends

Advantages Everyone gets something eventually Provides incentives to maintain contact Concerns Can complicate estimating long-run effects Care required with phase-in windows Do expectations of change actions today?

Round 1 Treatment: 1/2 Control: 1/2 Rotation design Round 2 Treatment from Round 1  Control —————————————————————————— Control from Round 1  Treatment Round 1 Treatment: 1/2 Control: 1/2

Groups get treatment in turns – Group A gets treatment in first period – Group B gets treatment in second period Advantages – Perceived as fairer; easier to get accepted Concerns – If people in Group B anticipate they’ll receive the treatment the next period, they can have a different behavior in the first period – Impossible to measure long-term impact since no control group after first period

Group Year 1Year 2Year 3 AGrade 3Grade 4Grade 3 BGrade 4Grade 3Grade 4 Schools in Varodara, India were divided into two groups: ( A and B) and teaching assistant offered in schools according to the following schedule:

 Sometimes it’s practically or ethically impossible to randomize program access  But most programs have less than 100% take- up  Randomize encouragement to receive treatment

Something that makes some folks more likely to use program than others Not itself a “treatment” For whom are we estimating the treatment effect? Think about who responds to encouragement Do not choose an encouragement that affects those who are different than the entire population

 Unit of Randomization  Ways to Randomize  Simple Randomization (eg: lottery)  Randomization “in the bubble”  Phase-in  Rotation  Encouragement Design  Multiple treatments, 2 stage  (if time) Stratification

Treatment 1 Treatment 2 Treatment 3 Multiple treatments

 Spillover: when the control group, although “untreated”, is affected (positively or negatively) by the treatment  Choose unit that contains spillover (ie randomize at school or village rather than individual level)  Measure Spillover: TUP (Bangladesh, Honduras, Peru, Pakistan, Ghana, Ethiopia, Yemen)

 Program which consists of targeting poorest families in a village and providing consumption support ($$ or food), asset transfer, livelihood training  Eventual graduation to microfinance

Communities (Total = 80) 40 treatment 40 control C 20 cont Households (Total = 1600) A 20 treat Ho do we determine impact? Program direct impact: A-C

Villagers may benefit from neighbors being treated Or they may be negatively affected Why would it be desirable (from a policy perspective) to measure spillover?

Communities (Total = 80) 40 treatment 40 control C 20 cont Households (Total = 2400) A 20 treat B 20 cont Ho do we determine impact? Program direct impact: A-C Program indirect impact: B-C

Objective: when you have a small sample, make sure key variables are balanced between Treatment and Control What is it: – dividing the sample into different subgroups – selecting treatment and control from each subgroup Stratify on variables that could have important impact on outcome variable (bit of a guess) Stratify on subgroups that you are particularly interested in (where may think impact of program may be different) Can get complex to stratify on too many variables Makes the draw less transparent the more you stratify

 Sampling!