Rerandomization to Improve Baseline Balance in Educational Experiments Kari Lock Morgan Department of Statistics Pennsylvania State University with Anna Saavedra and Amie Rapaport SREE March 1st, 2018

Motivation RCTs are the “gold standard” for estimating causal effects WHY? They eliminate confounding variables (balance covariates) They yield unbiased estimates … on average! For any particular experiment, covariate imbalance is possible (and likely!), and conditional bias exists

Typical RCT Randomize units to treatment groups Why not check balance before conducting the experiment, when you can still fix it? Conduct experiment Check baseline balance Analyze results

Rerandomization Collect covariate data Specify objective criteria for acceptable balance (Re)randomize units to treatment groups (Re)randomize units to treatment groups Randomize units to treatment groups Check balance unacceptable acceptable Conduct experiment Analyze results

Context Students learn Advanced Placement (AP) content through the Knowledge in Action (KIA) project-based learning approach designed to develop students’ deeper learning of skills and content RCT evaluation of KIA impact on student outcomes Recruited teachers across five districts, teachers in 76 schools enrolled Randomized at the school level within districts

KIA Covariates Only previous cohort data available at the time of randomization Covariates varied by district, but included Standardized test scores (PSAT/AP/8th grade) Socio-economic status % Nonwhite (some districts) Course (APES or APGOV) (some districts)

KIA Criteria Standardized difference in means: | 𝑋 1, 𝑇 − 𝑋 1, 𝐶 | 𝑠 1 , | 𝑋 2, 𝑇 − 𝑋 2, 𝐶 | 𝑠 2 , ... Thresholds varied by district: 0.05 – 0.25 Another option is Mahalanobis distance: 𝑿 𝑇 − 𝑿 𝑐 ′ cov 𝒙 −1 𝑿 𝑇 − 𝑿 𝑐

Covariate Balance: One District Percent reduction in variance: 𝑃𝑅𝐼𝑉= 𝑣𝑎𝑟 𝑥 𝑗, 𝑇 − 𝑥 𝑗, 𝐶 −𝑣𝑎𝑟 𝑥 𝑗, 𝑇 − 𝑥 𝑗, 𝐶 | 𝑟𝑒𝑟𝑎𝑛𝑑. 𝑣𝑎𝑟 𝑥 𝑗, 𝑇 − 𝑥 𝑗, 𝐶

Covariate Balance

Outcome Precision If PRIV is equal for all covariates, then PRIV for the outcome difference in means is 𝑃𝑅𝐼𝑉 𝑌 = 𝑅 2 × 𝑃𝑅𝐼𝑉 𝑋 Here, 𝑅 2 ≈0.75 and 𝑃𝑅𝐼𝑉 𝑋 ≈90%, so 𝑃𝑅𝐼𝑉 𝑌 ≈0.75×0.90=67.5% Precision increases by a factor of 1 1−0.675 = 3.1 Equivalent to more than tripling n!!! (Effective sample size goes from 76 to 234!) 76 to 293

More Power! Significance for smaller effect sizes! Use randomization test to take advantage of this; otherwise inference will be conservative

Regression Rerandomization reduces the need to account for covariates via modeling But if you do still choose to model… 𝑌 𝑖 =𝛼+𝜷 𝑿 𝑖 +𝜏 𝑊 𝑖 + 𝜀 𝑖 Better covariate balance: estimation of 𝜏 depends less on estimation of 𝜷… increases precision/power reduces reliance on modeling assumptions

Why Rerandomize? Avoid bad/unlucky randomizations Improve covariate balance Increase power Reduce reliance on assumptions

klm47@psu.edu Morgan, K.L., and Rubin, D.B. (2012). “Rerandomization to Improve Covariate Balance in Experiments,” Annals of Statistics, 40(2): 1262-1282. Morgan, K.L. and Rubin, D.B. (2015). “Rerandomization to Balance Tiers of Covariates,” JASA, 110(512): 1412 – 1421.