AADAPT Workshop South Asia Goa, December 17-21, 2009 Nandini Krishnan 1

 To Identify the causal effect of an intervention  separate the impact of the program from other factors  Need to find out what would have happened without the program  Cannot observe the same person with and without the program at the same point of time >> Create a valid counterfactual 2

High Yield Fertilizer Use OR ? 2) ? 1) ? High Yield Knowledge of Technologies Fertilizer Use

4 (+6) BIASED measure of the program impact

 Hard to distinguish causation from correlation from statistical analysis of existing data  However complex the statistics can only see that X moves with Y  Hard to correct for unobserved characteristics, like motivation/ability  May be very important- also affect outcomes of interest  Selection bias a major issue for impact evaluation  Projects started at specific times and places for particular reasons  Participants may be selected or self-select into programs  First farmers to adopt a new technology are likely to be very different from the average farmer, looking at their yields will give you a misleading impression of the benefits of a new technology 5

6 (+4) Impact of the program (+2) Impact of other (external) factors

 All those in the study have the same chance of being in the treatment or comparison group  By design, treatment and comparison have the same characteristics (observed and unobserved), on average  Only difference is treatment  With large sample, all characteristics average out  Unbiased impact estimates 7

 Lottery (0nly some receive)  Lottery to receive information about a new agricultural technology  Random phase-in (everyone gets it eventually)  Train some farmers groups each year  Variation in treatment  Some get information on new seed, others get credit, others get demonstration plot on their land etc  Encouragement design  One farmers support center per district  Some farmers get travel voucher to attend the center 8

Must receive the program Not suitable for the program Randomize who gets the program

 Budget constraint prevents full coverage  Random assignment (lottery) is fair and transparent  Limited implementation capacity  Phase-in gives all the same chance to go first  No evidence on which alternative is best  Random assignment to alternatives with equal ex ante chance of success 10

 Take up of existing program is not complete  Provide information or incentive for some to sign up- Randomize encouragement  Pilot a new program  Good opportunity to test design before scaling up  Operational changes to ongoing programs  Good opportunity to test changes before scaling them up 11

 Individual  Farm  Farmers’ Association  Irrigation block 12  Village level  Women’s association  Youth groups  School level

 If a program impacts a whole group-- usually randomize whole community to treatment or comparison  Easier to get big enough sample if randomize individuals Individual randomizationGroup randomization

 Randomizing at higher level sometimes necessary:  Political constraints on differential treatment within community  Practical constraints—confusing to implement different versions  Spillover effects may require higher level randomization  Randomizing at group level requires many groups because of within community correlation 14

15 Random assignment Treatment GroupControl Group Participants  Non-participants Evaluation sample Potential participants Maize producersRice producers Target population Commercial Farmers

 External validity  The evaluation sample is representative of the total population  The results in the sample represent the results in the population  We can apply the lessons to the whole population  Internal validity  The intervention and comparison groups are truly comparable   estimated effect of the intervention/program on the evaluated population reflects the real impact on that population 16

 An evaluation can have internal validity without external validity  Example: A randomized evaluation of encouraging women to stand for elections in urban areas may not tell us much about impact of a similar program in rural areas  An evaluation without internal validity, can’t have external validity  If you don’t know whether a program works in one place, then you have learnt nothing about whether it works elsewhere. 17

18 Random Sample- Randomization Randomization National Population Samples National Population

19 Stratification Randomization Population Population stratum Samples of Population Stratum

20 National Population Biased assignment USELESS! Randomization

 Efficacy  Proof of concept  Smaller scale  Pilot in ideal conditions  Effectiveness  At scale  Prevailing implementation arrangements -- “real life”  Higher or lower impact?  Higher or lower costs? 21

 Clear and precise causal impact  Relative to other methods  Much easier to analyze- Difference in averages  Cheaper (smaller sample sizes)  Easier to explain  More convincing to policymakers  Methodologically uncontroversial 22

Wheat plants do NOT  Raise ethical or practical concerns about randomization  Fail to comply with Treatment  Find a better Treatment  Move away—so lost to measurement  Refuse to answer questionnaires  Human beings can be a little more challenging!

 Some interventions can’t be assigned randomly  Partial take up or demand-driven interventions: Randomly promote the program to some  Participants make their own choices about adoption  Perhaps there is contamination- for instance, if some in the control group take-up treatment 24

 Those who get promotion are more likely to enroll  But who got promotion was determined randomly, so not correlated with other observables/non-observables  Compare average outcomes of two groups: promoted/not promoted  Effect of offering the encouragement (Intent-To- Treat)  Effect of the intervention on the complier population (Local Average Treatment Effect) ▪ LATE= ITT/proportion of those who took it up

Assigned to treatment Assigned to control DifferenceImpact: Average treatment effect on the treated Non-treated Treated Proportion treated 100%0%100% Impact of assignment 100% Mean outcome Intent-to-treat estimate 23/100%=23 Average treatment on the treated

Randomly Encouraged Not encouragedDifferenceImpact: Average treatment effect on compliers Non-treated (did not take up program) Treated (did take up program) Proportion treated 70%30%40% Impact of encouragement 100% Outcome Intent-to-treat estimate 8/40%=20 Local average treatment effect

 Calculating sample size incorrectly  Randomizing one district to treatment and one district to control and calculating sample size on number of people you interview  Collecting data in treatment and control differently  Counting those assigned to treatment who do not take up program as control—don’t undo your randomization!! 28

 The treatment already assigned and announced and no possibility for expansion of treatment  The program is over (retrospective)  Universal take up already  Program is national and non excludable  Freedom of the press, exchange rate policy (sometimes some components can be randomized)  Sample size is too small to make it worth it 29

30