1. Intervention Study: Kenya PRIMR Case Difference-in-Difference (DiD) Analysis
March 2017
Susan Edwards, RTI International
Sarrynna Sou, RTI International

2. Problem Overview
Goal: Determine the impact an intervention has on student reading performance. Ideal: Randomized Control Trial (RCT) - Treatment - Control Problem: Observational Study Non-Random Program Placement May Introduce Bias

3. Motivating Example – English Oral Reading Fluency
Question from Aims:
Did students’ English reading ability improve due to treatment?
How should we start to explore this question?
Determine the average fluency at baseline and endline.
Determine the average fluency at baseline and endline for the treatment group.
Determine the average fluency at baseline and endline for the treatment and control group.
October 2012
October 2013
Overall
45 wpm
50 wpm
-Open Topic_3_DiD.do File.
-Load the dataset. This is an appended dataset.
-Run ‘tab’ to describe the data.

4. Motivating Example – English Oral Reading Fluency
Question from Aims:
Did students’ English reading ability improve due to treatment?
How should we start to explore this question?
Determine the average fluency at baseline and endline.
Determine the average fluency at baseline and endline for the treatment group.
Determine the average fluency at baseline and endline for the treatment and control group.
October 2012
October 2013
Overall
45 wpm
50 wpm
Treatment Group
48 wpm
57 wpm
-Open Topic_3_DiD.do File.
-Load the dataset. This is an appended dataset.
-Run ‘tab’ to describe the data.

5. Motivating Example – English Oral Reading Fluency
Question from Aims:
Did students’ English reading ability improve due to treatment?
How should we start to explore this question?
Determine the average fluency at baseline and endline.
Determine the average fluency at baseline and endline for the treatment group.
Determine the average fluency at baseline and endline for the treatment and control group.
October 2012
October 2013
Overall
45 wpm
50 wpm
Treatment Group
48 wpm
57 wpm
Control Group
39 wpm
43 wpm
-Open Topic_3_DiD.do File.
-Load the dataset. This is an appended dataset.
-Run ‘tab’ to describe the data.

6. Motivating Example – English Oral Reading Fluency
October 2012
October 2013
Intervention Group
48 wpm
57 wpm
Problem:
Ignores general time trend
8.23
-Open Topic_3_DiD.do File.
-Load the dataset. This is an appended dataset.
-Run ‘tab’ to describe the data.
svy, over(cohort treat_phase): mean eq_orf

7. Motivating Example October 2012 October 2013 Intervention Group 48 wpm
Control Group
39 wpm
43 wpm
Problem:
Ignores pre-existing differences
13.82
svy, over(cohort treat_phase): mean eq_orf
9.34

8. Difference-in-Difference Analysis
Mimic an experimental research design using observational study data by studying the differential effect of a treatment on a “treatment group” versus a “control group” in a natural experiment.
8.23
svy, over(cohort treat_phase): mean eq_orf
3.75

9. Difference-in-Difference Analysis
Mimic an experimental research design using observational study data by studying the differential effect of a treatment on a “treatment group” versus a “control group” in a natural experiment.
Effect of Intervention –
taking into account pre-existing differences and general time trend. =
4.48
8.23
svy, over(cohort treat_phase): mean eq_orf

10. Difference-in-Difference Analysis
Effect of Intervention –
taking into account pre-existing differences and general time trend.
Key Assumption –
Whatever happened in the control group would also have happened in the treatment group if the intervention did not happen =
4.48
8.23
svy, over(cohort treat_phase): mean eq_orf

11. Difference-in-Difference Analysis – Practice with CLSPM
Calculate the Pre-Post Effect.
Calculate the General Trend over Time.
Calculate the Intervention Effect.
Identify what you think is post-intervention treatment vs. control effect. T2 E C T1
svy, over(cohort treat_phase): mean eq_orf C2 C1 D

12. Difference-in-Difference Analysis – Practice with CLSPM
Calculate the Pre-Post Effect. T2 – T1
Calculate the General Trend over Time. C2 – C1
Calculate the Intervention Effect. C - D = B - A = E
Identify what you think is post-intervention treatment vs. control effect. T2 E C T1
svy, over(cohort treat_phase): mean eq_orf C2 C1 D

13. DiD with STATA – English ORF Example
So we know the intervention improved students English reading ability by 4.48 wpm on average. But Is 4.48 wpm a significant impact?

14. DiD with STATA – English ORF Example
Easiest Way:
Regression Analysis
Must use the following variables:
Treatment Group Identifier (treatment)
Time Identifier (treat_phase)

15. DiD with STATA – English ORF Example
Model:
EQ_ORF = 𝛽 0 + 𝛽 1 (treatment) + 𝛽 2 (treat_phase)
+ 𝛽 3 (treatment)(treat_phase)
STATA Code:
svy: reg eq_orf i.treatment i.treat_phase
i.treatment#i.treat_phase
Independent Variables
What about the i?
i = categorical variable
Linear Regression
Run this STATA code.
Weighted Estimates
Dependent Variable

16. DiD with STATA – English ORF Example
svy: reg eq_orf i.treatment i.treat_phase treatment#treat_phase
What can we tell from this output?
Significant different between treatment and control groups.
Not a significant difference between baseline and endline.
DiD = 4.48

17. DiD with STATA – English ORF Example
Reference Cell Coding – Calculate Average ORF
Average ORF
Control - Baseline
38.98
Treatment - Baseline
= 48.32
Control – Endline
= 42.73
Treatment – Endline
= 56.55
Followed by hands on activity with Oral Reading Fluency.

18. DiD with STATA – English ORF Example
Reference Cell Coding – Calculate Average ORF
Average ORF
Control - Baseline
38.98
Treatment - Baseline
= 48.32
Control – Endline
= 42.73
Treatment – Endline
= 56.55
Followed by hands on activity with Oral Reading Fluency.

19. DiD with STATA – English ORF Example
4.48
9.34
svy, over(cohort treat_phase): mean eq_orf

20. DiD with STATA – English ORF Example
STATA Code:
svy: reg eq_orf i.treatment i.treat_phase treatment#treat_phase
Is 4.48 wpm a significant impact?
Followed by hands on activity with Oral Reading Fluency.
No. A 4.48 wpm increase is not statistically significant.

21. DiD with STATA – Practice with CLSPM
Is the intervention effect significant?
NOTE: I have not included effect sizes!

22. Particular Challenges with DiD and EGRA Data
Baseline Equivalences
Is it important that the treatment and control group start at the same point?
Key Assumption:
Whatever happened in the control group would also have happened in the treatment group if the intervention did not happen
Requires treatment group to be like the control group
Key things to note about the NSDUH study that makes it different from other studies
Include a slide about what the NSDUH is.

23. Particular Challenges with DiD and EGRA Data
Example 1: Imbalance in Demographics
60% of Treatment Group is made up of students that attended pre-school
20% of Control Group is made up of students that attended pre-school
Is this an issue?
Probably.
Depends on if we think this demographic affects our outcome of interest (reading ability).
Key things to note about the NSDUH study that makes it different from other studies
Include a slide about what the NSDUH is.

24. Particular Challenges with DiD and EGRA Data
Example 2: No Room to Grow
At baseline, the average ORF for grade 2 students is:
86.3 wpm in the treatment group
55.2 wpm in the control group
Is this an issue?
Most likely.
80+ wpm is extremely good for anyone
Treatment group may not be able to improve much.
Control group however has lots of room for growth.
Key things to note about the NSDUH study that makes it different from other studies
Include a slide about what the NSDUH is.

25. Particular Challenges with DiD and EGRA Data
Method for Detecting Imbalances
Balance Testing
Method for Correcting these Imbalances –
Re-weighting
Propensity Matching
Key things to note about the NSDUH study that makes it different from other studies
Include a slide about what the NSDUH is.

26. More Information
Susan Edwards Research Statistician
Sarrynna Sou Statistician