1. PolMeth2009: Freedman Panel Regression Adjustments to Experimental Data: Do David Freedman’s Concerns Apply to Political Science? Donald P. Green Yale University

2. Using covariates in the analysis of experimental results: the conventional view Benefit #1: addresses random imbalance Benefit #2: increases precision by reducing disturbance variance Drawback #1: burns up degrees of freedom Drawback #2: increases discretion, particularly in the absence of an ex ante analysis plan

3. Freedman’s critique of covariate adjustment Doesn’t follow from the experimental design Asymptotically unbiased but may be severely biased in finite samples Conventional regression estimates of standard errors may be severely biased

4. Freedman’s setup Assign a population of size n to treatment and control groups of size m and n-m, respectively Potential outcomes model, with responses that are deterministic functions of experimental assignments When assessing unbiasedness, consider the average estimate across all possible random assignment

5. Why the fuss? Experiments are becoming increasingly common, and covariate adjustment using regression is regarded as benign standard operating procedure Freedman’s claim that finite-sample bias is appreciable for n < 500 encompasses a large proportion of experimental studies published in political science

6. Aims of my paper Evaluate the magnitude of the bias for varying n Simulated data Real data (from experiments that have been reconfigured so that treatment and control are latent potential outcomes) Assess when biases are large and whether the symptoms of bias are detectable

7. Results Simulated examples: although it is possible to construct examples with severe biases, these tend to involve n<20 and noticeably heterogeneous treatment effects Analysis of actual experimental data shows very little bias in estimated treatment effects and fairly accurate estimated standard errors

8. Bottom line Freedman’s legacy is to challenge unreflective use of off-the-shelf statistical methods Regression is not unproblematic if applied to small populations with heterogeneous treatment effects, but now we have a clearer idea of what “small” means as a practical matter