Regression Discontinuity Design Saralyn Miller Southern Methodist University ____________________ Paper presented at the annual meeting of the Southwest Educational Research Association, San Antonio, TX, February 2-4,

Presentation Outline What is RDD? Example of RDD. History of RDD. Why was RDD developed? Assumptions of RDD. When to use RDD. How to know if discontinuity occurred. RDD limitations. Computing RDD in R. Conclusions. 2

What is RDD? RDD is an alternative to treatment/control experimental research. RDD is a quasi-experimental research design that allows the researcher to use a selection criteria rather than randomization of groups to determine treatment effects. RDD determines effectiveness of a program by comparing a treatment group that was selected according to the cutoff criterion to expected values from the comparison group that was not selected according to the cutoff criterion. 3

RDD Example Students are given the DIBELS ORF reading assessment. The students not meeting the benchmark requirements are placed in a tier 2 intervention (treatment group). Students above the benchmark are not given an intervention (control group). After the intervention is complete, students are post tested using DIBELS ORF. Control student scores are used to predict the scores of the students in the treatment condition if the intervention had not occurred. Actual treatment student scores are compared to the predicted value from control student scores. If the actual treatment student scores are statistically significantly different (slope and intercept) from the predicted values calculated from control student scores, a treatment effect is reported. 4

History of RDD RDD was first used by Campbell and Thistlewaite (1960). Developed for Psychology and Education. – Compensatory education programs (ex: Head Start). Campbell’s students, Sween and Trochim studied RDD further. – Non-linearity – Multiple cutoffs – Fuzzy discontinuity Several studies surfaced comparing RDD to experimental designs (ex: medical community). 5

Why was RDD developed? RDD was developed to control for selection bias. It provided an alternative to matching subjects in a T/C experimental design. – In experimental research, researchers attempt to equate groups on all other variables other than the actual treatment. Campbell and Stanley’s argument is that this is almost difficult to do. RDD is an experiment randomized at the cutoff value. – RDD allows for a more detailed description of the selection process. Randomizing would prove to be unethical or even political. 6

RDD Assumptions The Cutoff Criterion – Cutoff value must be followed. The Pre-Post Distribution (can’t be curved) – The relationship between the pretest and posttest can not be better explained with a curve. Comparison Group Pretest Variance – Comparison group must have a sufficient number of pretest values. Continuous Pretest Distribution – Both groups must come from a single continuous pretest distribution. Program Implementation – Program is delivered to all recipients in the same manner. 7

When do I use RDD? Pretest/posttest design – There are no matching treatment and control groups, but comparisons are made based on a predicted regression line. – If posttest scores for those in the treatment condition are better predicted by a new regression line than the regression line of the control group if it were extended, then a treatment effect is reported. Single subject research – Baseline serves as the control. – If intervention scores are better predicted by a new regression line than the regression line of the baseline if it were extended, then a treatment effect is reported. 8

How do I use RDD? Pretest/Posttest Design – The use of a separate control group is not idealistic. – Students are pre-tested and a cut-off criterion is used to determine treatment and control groups. (Example: students that do not meet a certain benchmark are put into a treatment condition.) – The intervention is implemented. – Students are post tested. 9

Continued – A regression model is calculated using pretest scores and a dummy-coded independent variable for treatment and control conditions to serve as predictors of posttest scores. 10

Continued Single Subject Research – Baseline serves as comparison data. – Time is your independent variable and once intervention begins, this time point is your cutoff criterion. – All data points after the cutoff criterion serve as your treatment data. – A regression model is calculated using time and a dummy- coded independent variable as a predictor of the dependent variable. 11

Continued – If a discontinuity is found at the cutoff criterion, then a treatment effect is reported. – A discontinuity can be either a statistically significant change in slope or y-intercept. 12

How do I know if there is a discontinuity? Compare slopes (interaction) and intercepts (main effects). – Treatment effects are reported if the regression line for the treatment group better predicts the score of the treatment students than if the control group predicted treatment student scores. If the interaction term shows statistical significance, the slope of the treatment group is SSD from the control group. If the main effect term for the dummy coded variable shows statistical significance, the intercept of the treatment group is SSD from the control group. 13

Continued 14

Limitations of RDD Number of subjects in the study needs to be almost 3 X the size of an experimental T/C design. One group is usually very small (usually those struggling or those exceeding) Power decreases. Some relationships that appear to be a discontinuity are actually better explained with a nonlinear line. Fuzzy discontinuity – this occurs when the cutoff criterion is not strictly adhered to. 15

Curvilinearity Problem (Trochim, 2006) 16

Steps for Computing Regression Discontinuity in R Subtract the cutoff score from the pretest value. Create a dummy coded variable for the 2 groups. Run a regression on post test scores given the new pretest scores and new dummy coded variable. Main effect of the dummy coded variable indicates SSD for the intercepts. Interaction effect indicates SSD for the slopes. 17

Example 1: Slope and Intercept are SSD pre<- c(3,4,7,8,9,12,15,17,18,19,7,5,10,12,16,17,15,4,9,16,22, 40,30,24,25,32,53,68,29,32,24,52,69,34,55,47,33,34,37,60,58,52,50,44,44) post<- c(6,5,14,12,20,30,35,40,41,44,20,10,20,33,29,34,40,12,20,43,30,45,30,45,36,44,53,75,40,41,34,55,72,40,55,50,40,4 1,42,65,65,55,52,47,49) preT<-ifelse(pre<=20,0,1) pre2<-pre-21 m1<-lm(post~pre2*preT) summary(m1) 18

Example 1: Slope and Intercept are SSD Call: lm(formula = post ~ pre2 * preT) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) < 2e-16 *** pre < 2e-16 *** preT e-09 *** pre2:preT e-10 *** --- Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: on 41 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 3 and 41 DF, p-value: < 2.2e-16 plot(m1) xyplot(post~pre*preT) plot(post~pre) abline(lm(post~pre), lwd=4) abline(lm(post[preT=="0"]~pre[preT=="0"]), col="blue",lwd=3) abline(lm(post[preT=="1"]~pre[preT=="1"]), col="red",lwd=3) abline(v=20) 19

Example 1: Slope and Intercept are SSD 20

Example 1: Slope and Intercept are SSD 21 par(mfrow=c(2,2)) plot(m1)

Example 1: Slope and Intercept are SSD 22 Pretest 1-20 Pretest >20

Example 2: Slope is SSD pre<- c(9,10,11,12,10,12,15,17,18,19,9,10,11,12,16,17,15,10,12,16,22, 40,30,24,25,32,53,68,29,32,24,52,69,34,55,47,33,34,37,60,58,52, 50,44,44) post<- c(15,16,17,18,17,19,26,27,28,30,20,21,20,25,29,23,25,17,20,29,3 0,45,30,45,36,44,53,75,40,41,34,55,72,40,55,50,40,41,42,65,65,5 5,52,47,49) preT<-ifelse(pre<=20,0,1) pre2<-pre-21 m1<-lm(post~pre2*preT) summary(m1) 23

Example 2: Slope is SSD Call: lm(formula = post ~ pre2 * preT) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) < 2e-16 *** pre e-06 *** preT pre2:preT * --- Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: on 41 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 3 and 41 DF, p-value: < 2.2e-16 24

Example 2: Slope is SSD 25

Example 2: Slope is SSD 26 Pretest 1-20 Pretest >20

Example 3: Intercept is SSD pre<- c(14,15,7,8,9,12,15,17,18,19,7,5,10,12,16,17,15,4,9,16,22,40,30,24,25,32,53,68,29,32,24,52,69,34,55,47,33,34,37,60,58,52,50,44,44) post<- c(64,64,63,66,63,65,66,66,67,68,63,62,65,66,66,60,61,62,59,59,4 7,52,49,55,47,50,59,65,50,51,45,51,57,49,51,58,49,54,52,60,60,6 1,60,58,49) preT<-ifelse(pre<=20,0,1) pre2<-pre-21 m1<-lm(post~pre2*preT) summary(m1) 27

Example 3: Intercept is SSD Call: lm(formula = post ~ pre2 * preT) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) < 2e-16 *** pre preT e-12 *** pre2:preT Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 3.12 on 41 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 3 and 41 DF, p-value: 2.806e-14 28

Example 3: Intercept is SSD 29

Example 3: Intercept is SSD 30 Pretest 1-20 Pretest >20

Example 4: Slope and Intercept are not SSD pre<- c(14,15,7,8,9,12,15,17,18,19,7,5,10,12,16,17,15,4,9,16,22,40,30,24,25,32,53,68,29,32,24,52,69,34,55,47,33,34,37,60,58,52,50,44,44) post<- c(44,44,43,46,43,45,46,46,47,48,43,42,45,46,46,47,45,45,48,49,4 7,52,49,55,47,50,59,65,50,51,45,51,57,49,51,58,49,54,52,60,60,6 1,60,58,49) preT<-ifelse(pre<=20,0,1) pre2<-pre-21 m1<-lm(post~pre2*preT) summary(m1) 31

Example 4: Slope and Intercept are not SSD lm(formula = post ~ pre2 * preT) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) <2e-16 *** pre preT pre2:preT Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: on 41 degrees of freedom Multiple R-squared: 0.784, Adjusted R-squared: F-statistic: on 3 and 41 DF, p-value: 1.052e-13 32

Example 4: Slope and Intercept are not SSD 33

Example 4: Slope and Intercept are not SSD 34 Pretest 1-20 Pretest >20

Justin maximize<-read.table("C://Users/sjmiller/Desktop/RDD Justin.txt",header=T) attach(maximize) library(MASS) library(lattice) Book2<-ifelse(Book==0, 0, 1) Time2<-Time-11 m1<-lm(WRC~Time2*Book2, na.action=na.omit) summary(m1) 35

Call: lm(formula = WRC ~ Time2 * Book2, na.action = na.omit) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) Time Book * Time2:Book Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: on 53 degrees of freedom (16 observations deleted due to missingness) Multiple R-squared: , Adjusted R-squared: F-statistic: 174 on 3 and 53 DF, p-value: < 2.2e Justin

par(mfrow=c(2,2)) plot(m1) 37 Justin

plot(m1) plot(WRC~Time) abline(lm(WRC~Time), lwd=4) abline(lm(WRC[Book=="0"]~Time[Book=="0"]), col="blue",lwd=3) abline(lm(WRC[Book>="1"]~Time[Book>="1"]), col="red", lwd=3) abline(v=10) 38 Justin

Kristen maximize<-read.table("C://Users/sjmiller/Desktop/RDD Kristen.txt",header=T) attach(maximize) Book2<-ifelse(Book.1==0, 0, 1) Time2<-Time.1-22 m2<-lm(WRC.1~Time2*Book2, na.action=na.omit) summary(m2) 39

Call: lm(formula = WRC.1 ~ Time2 * Book2, na.action = na.omit) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) e-05 *** Time Book e-05 *** Time2:Book * --- Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: on 50 degrees of freedom (19 observations deleted due to missingness) Multiple R-squared: , Adjusted R-squared: F-statistic: on 3 and 50 DF, p-value: < 2.2e Kristen

plot(m2) plot(WRC.1~Time.1) abline(lm(WRC.1~Time.1), lwd=4) abline(lm(WRC.1[Book.1=="0"]~Time.1[Book.1=="0"]), col="blue",lwd=3) abline(lm(WRC.1[Book.1>="1"]~Time.1[Book.1>="1"]), col="red", lwd=3) abline(v=21) 41 Kristen

par(mfrow=c(2,2)) plot(m2) 42 Kristen

Grace maximize<-read.table("C://Users/sjmiller/Desktop/RDD Grace.txt",header=T) attach(maximize) Book2<-ifelse(Book.2==0, 0, 1) Time2<-Time.2-18 m3<-lm(WRC.2~Time2*Book2, na.action=na.omit) summary(m3) 43

Call: lm(formula = WRC.2 ~ Time2 * Book2, na.action = na.omit) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) e-11 *** Time * Book ** Time2:Book Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 4.78 on 59 degrees of freedom (10 observations deleted due to missingness) Multiple R-squared: , Adjusted R-squared: F-statistic: on 3 and 59 DF, p-value: < 2.2e Grace

plot(m3) plot(WRC.2~Time.2) abline(lm(WRC.2~Time.2), lwd=4) abline(lm(WRC.2[Book.2=="0"]~Time.2[Book.2=="0"]), col="blue",lwd=3) abline(lm(WRC.2[Book.2>="1"]~Time.2[Book.2>="1"]), col="red", lwd=3) abline(v=17) 45 Grace

Conclusions: RDD RDD is an alternative to experimental research when a control group is not accessible. Provides an alternative approach when selection bias is prevalent. Treatment effects are reported if either the intercepts or slopes are statistically significantly different. 46

References Cook, T. D. (2008). “Waiting for life to arrive”: A history of the regression- discontinuity design in psychology, statistics and economics. Journal of Econometrics, 142, Trochim, W. M. K. (1984). Research design for program evaluation: The regression- discontinuity approach. Sage, Beverly Hills, CA. Trochim, W. M. K. (2007). Regression-discontinuity analysis. Retrieved April 10, 2010, from 47