1. Threats and Analysis

2. Course Overview What is evaluation?
Measuring impacts (outcomes, indicators)
Why randomize?
How to randomize?
Sampling and sample size
Threats and Analysis
Scaling Up
Project from Start to Finish

3. Lecture Overview Attrition Spillovers
Lecture Overview
Attrition
Spillovers
Partial Compliance and Sample Selection Bias
Intention to Treat & Treatment on Treated
Choice of Outcomes
External Validity 3

4. Attrition

5. Attrition Bias: An Example
Attrition Bias: An Example
The problem you want to address:
Some children don’t come to school because they are too weak (undernourished)
You start a feeding program where free food vouchers are delivered at home in certain villages and want to do an evaluation
You have a treatment and a control group
Weak, stunted children start going to school more if they live in a treatment village
First impact of your program: increased enrollment.
In addition, you want to measure the impact on child’s growth
Second outcome of interest: Weight of children
You go to all the schools (in treatment and control districts) and measure everyone who is in school on a given day
Will the treatment-control difference in weight be over-stated or understated? 5

6. 6

7. 7

8. Attrition
Is it a problem if some of the people in the experiment vanish before you collect your data?
It is a problem if the type of people who disappear is correlated with the treatment.
Why is it a problem?
When should we expect this to happen? 8

9. What if only children > 21 Kg come to school?
Will you underestimate the impact?
Will you overestimate the impact?
Neither
Ambiguous
Don’t know

10. What if only children > 21 Kg come to school?10

11. When is attrition not a problem?
When it is less than 25% of the original sample
When it happens in the same proportion in both groups
When it is correlated with treatment assignment
All of the above
None of the above

12.
Attrition Bias
Devote resources to tracking participants after they leave the program
If there is still attrition, check that it is not different in treatment and control. Is that enough?
Also check that it is not correlated with observables.
Try to bound the extent of the bias
Suppose everyone who dropped out from the treatment got the lowest score that anyone got; suppose everyone who dropped out of control got the highest score that anyone got…
Why does this help? 12

13. Spillovers

14. What Else Could Go Wrong?
Total
Population
Target Population
Not in evaluation
Evaluation Sample
Treatment Group
Random Assignment
Control Group 14

15. Spillovers, Contamination
Total
Population
Target Population
Not in evaluation
Treatment 
Evaluation Sample
Treatment Group
[Repeat from Lecture 2]
[First click:]
[Second Click]: Through random assignment, we can ensure the study is internally valid
[Third Click]: even though there will be some participants and some non-participants within our treatment group, at the first stage, we restrict our analysis to comparing the entire treatment group with the entire control group
There is a saying: Once a treatment group, always treatment group
[Fourth Click]
Similarly: once a control group, always a control group
[FifthClick]
Highlight No-Shows and Crossovers. This is called non compliance
We will cover how to account for this later.
Random Assignment
Control Group 15

16. Spillovers, Contamination
Total
Population
Target Population
Not in evaluation
Treatment 
Evaluation Sample
Treatment Group
It’s worth noting that spillovers won’t always lead you to under-estimate your impact
Random Assignment
Control Group 16

17. Example: Vaccination for Chicken Pox
Example: Vaccination for Chicken Pox
Suppose you randomize chicken pox vaccinations within schools
Suppose that prevents the transmission of disease, what problems does this create for evaluation?
Suppose spillovers are local? How can we measure total impact? 17

18. Spillovers Within School
Spillovers Within School
Without Spillovers
School A
Treated?
Outcome
Pupil 1
Yes
no chicken pox
Total in Treatment with chicken pox
Pupil 2 No
chicken pox
Total in Control with chicken pox
Pupil 3
Pupil 4
Treament Effect
Pupil 5
Pupil 6
With Spillovers
Suppose, because prevalence is lower, some children are not re-infected with chicken pox
Treatment Effect 18

19. Spillovers Within School 0%
100%
-100%
67%
- 6767%
Spillovers Within School
Without Spillovers
School A
Treated?
Outcome
Pupil 1
Yes
no chicken pox
Total in Treatment with chicken pox
Pupil 2 No
chicken pox
Total in Control with chicken pox
Pupil 3
Pupil 4
Treament Effect
Pupil 5
Pupil 6
With Spillovers
Suppose, because prevalence is lower, some children are not re-infected with chicken pox
Treatment Effect
- 67% 19

20. How to Measure Program Impact in the Presence of Spillovers?
How to Measure Program Impact in the Presence of Spillovers?
Design the unit of randomization so that it encompasses the spillovers
If we expect spillovers that are all within school:
Randomization at the level of the school allows for estimation of the overall effect 20

21. Partial Compliance and Sample Selection Bias

22. Non Compliers What can you do? Can you switch them? No!
Target Population
Not in evaluation
What can you do?
Can you switch them?
No!
Evaluation Sample
Participants
Treatment group
Random Assignment
So sometimes, you intend for there to be complete coverage in your treatment group, and none in your control, but people don’t always comply with their original assignment
No-Shows
Control group
Non-Participants
Cross-overs 22

23. Non Compliers What can you do? Can you drop them? No!
Target Population
Not in evaluation
What can you do?
Can you drop them?
No!
Evaluation Sample
Participants
Treatment group
Random Assignment
[Repeat from Lecture 2]
[First click:]
[Second Click]: Through random assignment, we can ensure the study is internally valid
[Third Click]: even though there will be some participants and some non-participants within our treatment group, at the first stage, we restrict our analysis to comparing the entire treatment group with the entire control group
There is a saying: Once a treatment group, always treatment group
[Fourth Click]
Similarly: once a control group, always a control group
[FifthClick]
Highlight No-Shows and Crossovers. This is called non compliance
We will cover how to account for this later.
No-Shows
Control group
Non-Participants
Cross-overs 23

24. Non Compliers You can compare the original groups Target Population
Not in evaluation
You can compare the
original groups
Evaluation Sample
Participants
Treatment group
Random Assignment
No-Shows
Control group
Non-Participants
Cross-overs 24

25.
Sample Selection Bias
Sample selection bias could arise if factors other than random assignment influence program allocation
Even if intended allocation of program was random, the actual allocation may not be 25

26.
Sample Selection Bias
Individuals assigned to comparison group could attempt to move into treatment group
School feeding program: parents could attempt to move their children from comparison school to treatment school
Alternatively, individuals allocated to treatment group may not receive treatment
School feeding program: some students assigned to treatment schools bring and eat their own lunch anyway, or choose not to eat at all. 26

27. Intention to Treat & Treatment on Treated

28. ITT and ToT Vaccination campaign in villages
ITT and ToT
Vaccination campaign in villages
Some people in treatment villages not treated
78% of people assigned to receive treatment received some treatment
What do you do?
Compare the beneficiaries and non-beneficiaries?
Why not? 28

29. Intention to Treat (ITT)
Intention to Treat (ITT)
What does “intention to treat” measure?
“What happened to the average child who is in a treated school in this population?”
Is this difference the causal effect of the intervention? 29

30. 30

31. 3 0.9 2.1 Intention School 1 to Treat ? Treated? Pupil 1 yes 4 Pupil 2 3
0.9
2.1
Intention
School 1
to Treat ?
Treated?
Pupil 1
yes 4
Pupil 2
Pupil 3
Pupil 4 no
Pupil 5
Pupil 6 2
Pupil 7
Pupil 8 6
School 1:
Pupil 9
Avg. Change among Treated
(A)
Pupil 10
School 2:
Avg. Change among Treated A=
Avg. Change among not-treated
(B)
School 2
A-B 1
Avg. Change among Not-Treated B=
Observed
Change in
weight 31

32. From ITT to Effect of Treatment on the Treated (TOT)
From ITT to Effect of Treatment on the Treated (TOT)
The point is that if there is leakage across the groups, the comparison between those originally assigned to treatment and those originally assigned to control is smaller
But the difference in the probability of getting treated is also smaller
Formally, we obtain the probability of being induced into getting treatment by “instrumenting” the probability of treatment by the original assignment 32

33. Estimating TOT What values do we need? Y(T) Y(C) Prob[treated|T]
Prob[treated|C]

34. Treatment on the Treated (TOT)
Treatment on the Treated (TOT)
𝐵= 𝐸 𝑌 𝑖 𝑧 𝑖 =1 −𝐸 𝑌 𝑖 𝑧 𝑖 =0 𝐸 𝑠 𝑖 𝑧 𝑖 =1 −𝐸 𝑠 𝑖 𝑧 𝑖 =0 𝑌 𝑇 −𝑌 𝐶 𝑃𝑟𝑜𝑏 𝑡𝑟𝑒𝑎𝑡𝑒𝑑 𝑇 −𝑃𝑟𝑜𝑏[𝑡𝑟𝑒𝑎𝑡𝑒𝑑|𝐶] 34

35.
TOT Estimator 35

36. TOT Estimator 3 5.25 Intention School 1 to Treat ? Treated? Pupil 1
TOT Estimator 3
0.9
60%
20%
2.1
40%
5.25
Intention
School 1
to Treat ?
Treated?
Pupil 1
yes 4
Pupil 2
Pupil 3
A =
Gain if Treated
Pupil 4 no
B =
Gain if not Treated
Pupil 5
Pupil 6 2
Pupil 7
ToT Estimator: A-B
Pupil 8 6
Pupil 9
Pupil 10
A-B =
Y(T)-Y(C)
Avg. Change Y(T)=
Prob(Treated|T)-Prob(Treated|C)
School 2
Y(T) 1
Y(C)
Prob(Treated|T)
Prob(Treated|C)
Avg. Change Y(C) =
A-B
Observed
Change in
weight 36

37. ITT vs TOT If obtaining estimate is easy, why not always use TOT?
ITT vs TOT
If obtaining estimate is easy, why not always use TOT?
TOT estimates Local Average Treatment Effect (LATE)
ITT may be policy relevant parameter of interest
For example, we may not be interested in the medical effect of deworming treatment, but what would happen under an actual deworming program.
If students often miss school and therefore don't get the deworming medicine, the intention to treat estimate may actually be most relevant. 37

38. Choice of outcomes

39. Multiple outcomes How do we decide on which outcomes to focus on?
Multiple outcomes
How do we decide on which outcomes to focus on?
Only outcomes that are statistically significantly different?
The more outcomes you look at, the higher the chance you find at least one significantly affected by the program
How do you account for this?
Pre-specify outcomes of interest (JPAL-AEA web registry)
Report results on all measured outcomes, even null results
Correct statistical tests (Bonferroni) 39

40. External validity

41. Threat to External Validity:
Threat to External Validity:
Behavioral responses to evaluations
Generalizability of results 41

42. Threat to external validity: Behavioral responses to evaluations
Threat to external validity: Behavioral responses to evaluations
One limitation of evaluations is that the evaluation itself may cause the treatment or comparison group to change its behavior
Treatment group behavior changes: Hawthorne effect
Comparison group behavior changes: John Henry effect
Minimize salience of evaluation as much as possible
Consider including controls who are measured at end-line only 42

43. Generalizability of Results
Generalizability of Results
Depend on three factors:
Program Implementation: can it be replicated at a large (national) scale?
Study Sample: is it representative?
Sensitivity of results: would a similar, but slightly different program, have same impact? 43

44. Further Resources
Using Randomization in Development Economics Research: A Toolkit (Duflo, Glennerster, Kremer)
Mostly Harmless Econometrics (Angrist and Pischke)
Identification and Estimation of Local Average Treatment Effects (Imbens and Angrist, Econometrica, 1994).
(LATE via random assignment, first stage, exclusion, and monotonicity)