Using A Regression Discontinuity Design (RDD) to Measure Educational Effectiveness: Howard S. Bloom

This Talk Will: introduce the history and logic of RDD, introduce the history and logic of RDD, consider conditions for its internal validity, consider conditions for its internal validity, considers its sample size requirements, considers its sample size requirements, consider its dependence on functional form, consider its dependence on functional form, illustrate some specification tests for it, illustrate some specification tests for it, consider limits to its external validity, consider limits to its external validity, consider how to deal with noncompliance, consider how to deal with noncompliance, describe an application. describe an application.

RDD History In the beginning there was Thislethwaite and Campbell (1960) In the beginning there was Thislethwaite and Campbell (1960) This was followed by a flurry of applications to Title I (Trochim, 1984) This was followed by a flurry of applications to Title I (Trochim, 1984) Only a few economists were involved initially (Goldberger, 1972) Only a few economists were involved initially (Goldberger, 1972) Then RDD went into hibernation Then RDD went into hibernation It recently experienced a renaissance among economists (e.g. Hahn, Todd and van der Klaauw, 2001; Jacob and Lefgren, 2002) It recently experienced a renaissance among economists (e.g. Hahn, Todd and van der Klaauw, 2001; Jacob and Lefgren, 2002) Tom Cook has written about this story Tom Cook has written about this story

RDD Logic Selection on an observable (a rating) Selection on an observable (a rating) A tie-breaking experiment A tie-breaking experiment Modeling close to the cut-point Modeling close to the cut-point Modeling the full distribution of ratings Modeling the full distribution of ratings

RDD As A Linear Regression

Conditions for Internal Validity The outcome-by-rating regression is a continuous function (absent treatment). The outcome-by-rating regression is a continuous function (absent treatment). The cut-point is determined independently of knowledge about ratings. The cut-point is determined independently of knowledge about ratings. Ratings are determined independently of knowledge about the cut-point. Ratings are determined independently of knowledge about the cut-point. The functional form of the outcome-by- rating regression is specified properly. The functional form of the outcome-by- rating regression is specified properly.

RDD Statistical Model where: Y i = outcome for subject i, T i = one for subjects in the treatment group and zero otherwise, R i = rating for subject i, e i = random error term for subject i, which is independently and identically distributed

Variance of the Impact Estimator  2 = variance of mean outcomes across subjects in the treatment group or comparison group treatment group or comparison group R 1 2 = square of the correlation between outcomes and ratings within the treatment and comparison group ratings within the treatment and comparison group R 2 2 = square of the correlation between treatment status and the rating the rating P = P = proportion of subjects in the treatment group, N = total number of subjects

Sample Size Implications Because of the substantial multi-collinearity that exists between its rating variable and treatment indicator, an RDD requires 3 to 4 times as many sample members as a corresponding randomized experiment Because of the substantial multi-collinearity that exists between its rating variable and treatment indicator, an RDD requires 3 to 4 times as many sample members as a corresponding randomized experiment

Specification Tests Using the RDD to compare baseline characteristics of the treatment and comparison groups Using the RDD to compare baseline characteristics of the treatment and comparison groups Re-estimating impacts and sequentially deleting subjects with the highest and lowest ratings Re-estimating impacts and sequentially deleting subjects with the highest and lowest ratings Re-estimating impacts and adding: Re-estimating impacts and adding:  a treatment status/rating interaction  a quadratic rating term  interacting the quadratic with treatment status Using non-parametric estimation Using non-parametric estimation

External Validity Estimating impacts at the cut-point Estimating impacts at the cut-point Extrapolating impacts beyond the cut-point with a simple linear model Extrapolating impacts beyond the cut-point with a simple linear model Estimating varying impacts beyond the cut- point with more complex functional forms Estimating varying impacts beyond the cut- point with more complex functional forms

Dealing With Noncompliance Sharp and fuzzy RDDs Sharp and fuzzy RDDs No-shows and crossovers No-shows and crossovers The effect of intent to treat (ITT) The effect of intent to treat (ITT) The local average treatment effect (LATE) The local average treatment effect (LATE) The effect of treatment on the treated (TOT) The effect of treatment on the treated (TOT) Where r T and r C = the proportion of the treatment and control groups receiving treatment, respectively

Application of RDD To Reading First Reading First (RF) is a cornerstone of No Child Left Behind Reading First (RF) is a cornerstone of No Child Left Behind RF resources are allocated purposefully to schools that need it most and will benefit most RF resources are allocated purposefully to schools that need it most and will benefit most Some districts allocated RF resources based on quantitative indicators Some districts allocated RF resources based on quantitative indicators We chose a sample of 251 schools near the cut- points for 17 such districts and 1 state We chose a sample of 251 schools near the cut- points for 17 such districts and 1 state

References Cook, T. D. (in press) “Waiting for Life to Arrive: A History of the Regression-discontinuity Design in Psychology, Statistics and Economics” Journal of Econometrics. Goldberger, A. S. (1972) “Selection Bias in Evaluating Treatment Effects: Some Formal Illustrations” (Discussion Paper , Madison WI: University of Wisconsin, Institute for Research on Poverty, June). Hahn, H., P. Todd and W. van der Klaauw (2001) “Identification and Estimation of Treatment Effects with a Regression-Discontinuity Design” Econometrica, 69(3): 201 – 209. Jacob, B. and L. Lefgren (2004) “Remedial Education and Student Achievement: A Regression-Discontinuity Analysis” Review of Economics and Statistics, LXXXVI.1: Thistlethwaite, D. L. and D. T. Campbell (1960) “Regression Discontinuity Analysis: An Alternative to the Ex Post Facto Experiment” Journal of Educational Psychology, 51(6): 309 – 317. Trochim, W. M. K. (1984) Research Designs for Program Evaluation: The Regression-Discontinuity Approach (Newbury Park, CA: Sage Publications).