povertyactionlab.org How to Randomize? Abhijit Banerjee Massachusetts Institute of Technology

povertyactionlab.org Different types of randomization

Randomization Designs Lottery design Phase-in design Encouragement design

povertyactionlab.org What should we worry about when we randomize?

Key constraints Cannot interfere significantly with program operations Cannot be perceived as unfair Must be politically feasible Must be ethical

One Trick: Limited Resources Many programs only have so much resources Many more people are eligible than they could serve Examples: –Training program –Limited textbooks to distribute How to allocate resources?

Lottery design Randomly choose from applicant pool In this design participants are explicitly told that “winners” and “losers” will end up with different outcomes Use when there is no good reason to discriminate among a subset of applicants Lottery is perceived as fair Transparent Politically more feasible often

povertyactionlab.org What are other types of lotteries?

Lottery Design Lottery need not be over individuals Can be over groups Which micro-credit groups? Can be over communities Which villages should we enter? Can be over schools Balsakhi

A lottery over program sites Works if there are multiple sites How are the program sites chosen? Some choices arbitrary? What is the affected population?

Ethicality and Political Feasibility Transparency often helps Can we give something to the control sites? What happened in the balsakhi intervention?

povertyactionlab.org What to do if program has 500 applicants for roughly 500 slots? Or less?

A less obvious lottery design One trick: Advertise program and greatly increase applicants –Now lottery is feasible Is this ethical? Would it be ethical we added slots to the program and only gave these to the new applicants? Or desirable from the evaluation point of view?

povertyactionlab.org When is a lottery impossible?

Where a lottery fails Suppose there are 2000 applicants. Screening of applications/need produces 500 “worthy” candidates Lottery infeasible

povertyactionlab.org What can be done in this situation?

What can be done? Potential Solution: –What are they screening for? –What elements are essential? –What elements are arbitrary? Example: Training program –2000 candidates –1000 fit criteria such as poor enough to be worthy, qualified enough to take advantage of training –250 chosen of out of the 1000 based on particular attributes –NGO willing to allocate remaining 250 by lottery Lesson: Many parts of a selection rule are designed simply to weed out candidates

povertyactionlab.org Can we think of any other trick that will work here?

Phase-in Design In this design everyone is told that they will end up with the same outcome but some later than others. Use the fact that the program going to expand Example: –In 5 years, 500 schools will be covered. What determines which schools will be covered? –Some choices may be based on need, potential impact, etc. –Some choices largely arbitrary We can therefore choose who gets program early at random –Do this on population about which choices are arbitrary What is the affected population? How are the comparisons made?

povertyactionlab.org Yet another trick?

Encouragement design In this design everyone is immediately eligible to receive the program—there is enough funding to cover all of them. However not all of them will necessarily take it up Pick some people at random and encourage them to use program Use non-encouraged as control Ethical? What population is this a treatment effect on?

povertyactionlab.org Level of Randomization

The unit of randomization Can be: –Individual: A person/A household/A selected group of people –Collective: A community/An institution/ A village

povertyactionlab.org What are the key concerns in this decision?

Key concerns Political feasibility Practicality of implementation Scope of the impact Data collection costs

Individual versus collective randomization Pros of individual level randomization: –Data collection is easier: counterexample? –We can measure the impact of individual characteristics on what the program does Cons: –The program may naturally operate at another level In the balsakhi case how would we do an individual level randomization? –May create conflict within a group/community –Even if the treatment is on the individual the effect may go beyond him.

Randomize evenly? Suppose 100 slots are available in a training program. Two hundred men and women apply You are able to randomize these 100 slots One strategy: –Give each person a lottery number 1 to 200. –Randomly pick 100 numbers. –Those people are chosen Anything wrong with this strategy?

A Simplified India For simplification, we reference only the sixteen largest Indian states These 16 states are divided into four geographic regions We also assume there are exactly 20 districts in each Indian state, implying 320 districts in all of India.

Regions of India Northern States Himachal Pradesh Haryana Punjab Uttar Pradesh Western States Gujarat Madhya Pradesh Maharashtra Rajasthan Southern States Andhra Pradesh Karnataka Kerala Tamil Nadu Eastern States Assam Bihar Orissa West Bengal

Experiment Design For Program X, we plan to run an intervention with 40 treatment districts and 40 control districts. These districts are randomly selected from the entire pool of 320 Indian districts. The composition of the treatment group, broken down by geographic region, varies considerably each time we randomize the sample.

Randomized Placement of Treatment Groups Randomize

Randomized Placement of Treatment Groups Randomize

povertyactionlab.org Why is this a problem?

Stratification Divide India into four regions of 80 districts each. Randomly choose 10 control and 10 treatment in each. Why not 70 in control?

Problems with unbalanced treatment and control groups Shocks hitting some regions but not others. Suppose you wanted to know the treatment effect in each region?

Advantages of Stratification Good control groups. Ensure that each area is represented in treatment and control –Important if some shocks are hitting men or women specifically Allows estimation of treatment effects for each group –Important if you care about differential treatment for men and women

povertyactionlab.org Would estimates be biased if there were no stratification? No. Stratification increases power

Variables to stratify on Context specific Groups affected by different shocks –Different regions –Different income groups –Different occupations Groups with different treatment effects What happened in the balsakhi case?

povertyactionlab.org Poverty Action Lab: Translating Research into Action MIT