Applied Research Seminar Public Policy Research Center REGRESSION DISCONTINUITY DESIGN CAN BE YOUR FRIEND: DEVELOPING EVIDENCE IN THE REAL WORLD David Kimball Adriano Udani Department of Political Science, UMSL RD Design and Applications 1

 Purpose  Scholarship and Contributions  Design  Application RD Design and Applications 2 AGENDA

 Method to estimate treatment effects in natural setting  Observed continuous variable and causal variable of interest exhibit a discontinuous increase at a certain threshold  Address confounding factors influencing control and treatment  Empirically verify assumptions  strengthens internal validity  Applies to observations only near “cutoff point”  limits external validity REGRESSION DISCONTINUITY DESIGN RD Design and Applications 3

 Impact of scholarships on future academic outcomes  Awards based on test scores, measured against cutoff point (c)  If score > c, then individual receive an award  Estimated treatment effect applies to individuals near the cutoff point  Assume these individuals have similar characteristics  EXCEPT receipt of award THISTLEWAITE AND CAMPBELL (1960) RD Design and Applications 4

Source of RDThresholdTreatmentOutcome PerformanceTest score Team standings Scholarship Relegation Achievement Club revenues PopulationCity/County SizeFederal funds Election rules Official salaries Voting behavior Turnout Candidate entry Size thresholdSchool size Firm size Class size Anti-bias law Achievement Productivity Eligibility criteriaCity poverty rank Prisoner index Anti-poverty program High security Voting behavior Recidivism Age thresholdVoting age Student birth month Past voting Years of education Turnout Earnings Close electionsVote majorityIncumbencyPolitician behavior IncomeAnnual incomeHealth insuranceHealth TYPES OF RD STUDIES RD Design and Applications 5

 Health  Low birth weight babies (Almond et al. 2010)  Young adults who lose health insurance (Anderson et al. 2012)  Education  U.S. School Bond Referenda (Cellini, Ferreira, and Rothstein 2010)  Management studies  Yelp.com ratings (Anderson and Magruder 2012; Lucas 2012)  Political Science  Split tickets in the Senate (Butler and Butler 2005)  Incumbency effect (Snyder 2005) *  Coattails of Members of Congress (Broockman 2009)  U.K. House of Commons (Eggers and Hainmueller 2009)  Close House Races (Caughey and Sekhon 2011)  U.S. mayoral races (Gerber and Hopkins 2011) STUDIES THAT USE RDD RD Design and Applications 6

RDD: TREATMENT EFFECT Source: Perraillon (2013): RD Design and Applications 7

BLACK MAYORS HIRE MORE BLACK POLICE Source: Hopkins and McCabe 2012 RD Design and Applications 8

 Be wary of RD design if there is strategic behavior or manipulation near threshold.  Information  Incentives  Capacity EVALUATE RD ASSUMPTION THEORETICALLY RD Design and Applications 9

 Balance test: Plotting means of pre-treament covariates in control group vs. treatment group (difference of means).  Density test: Examine distribution of observations just above and just below threshold.  Test causal direction (outcome or treatment DOES NOT predict pre-treatment DV or other covariates)  Placebo test: Look for other discontinuities in the range of scores. TEST RD ASSUMPTIONS EMPIRICALLY RD Design and Applications 10

 Test different specifications.  Linear  Polynomial  Local regression  Test different “discontinuity samples” (different bandwidths).  Test sensitivity to inclusion of pretreatment covariates CHECK STABILITY OF RD RESULTS RD Design and Applications 11

MISSOURI SCHOOLS APPLICATION RD Design and Applications 12

 RD design is an appealing form of natural experiment.  Weak assumptions compared to other empirical methods  In many cases the assumptions are plausible  Policymakers might consider a threshold for policy applications – this would favor empirical analysis of the policy’s impact. RD Design and Applications 13 IMPLICATIONS FOR POLICY ANALYSIS

 Joshua D. Angrist and Jörn-Steffen Pischke, Mostly Harmless Econometrics (Princeton University Press, 2008).  Thad Dunning, Natural Experiments in the Social Sciences (Cambridge University Press, 2012).  Andrew C. Eggers, Anthony Fowler, Jens Hainmueller, Andrew B. Hall, and James M. Snyder, Jr “On the Validity of the Regression Discontinuity Design for Estimating Electoral Effects: New Evidence from over 40,000 Close Races.” American Journal of Political Science (May 2014). RD Design and Applications 14 REFERENCES