1. sources of bias in experiments and quasi-experiments sean f. reardon stanford university 11 december, 2006

2. three populations  population of interest (POI): the population for whom we would like to estimate the average treatment effect  population of study (POS): the population of whom the study sample is representative  population of causal inference (POC): the population for whom we can make a causal inference

3. tradeoffs between bias and generalizability  we want to estimate the average treatment effect () in some population of interest (POI).  tells us how much would should we expect the outcome Y of a person randomly chosen from P to differ depending on whether we assign them to T or C.  but we only estimate  in POC  bias arises when POC is not the same as POI: bias 1:  POC is not the same as  POS bias 2:  POS is not the same as  POI

4. strategies for minimizing bias  randomized experiments: we get unbiased estimate of  POC and  POC = POS, so RCT eliminates bias 1 but bias 2 may be large or small (external validity)  regression discontinuity: we get unbiased estimate of  POC (under weak assumptions) but at the cost of making  POC ≠  POS POC is generally small (POC is the region of the population near the discontinuity) relative to both POS and POI but sometimes the region around discontinuity is the population of interest

5. strategies for minimizing bias (cont.)  matching (including fixed effects): attempts to get unbiased estimate of  POC through matching but at the cost of making POC smaller relative to POS (and POI), because matching allows estimation of treatment effect only in region of common support but observational matching studies are easier to do with sample of the population of interest than are experiments, so bias 2 may be smaller in matching if region of common support (POC) approximates POI. fixed effects a form of matching (matching on invariant observed or unobserved factors)

6. How well do quasi-experimental methods do at eliminating bias?  Shadish & Clark paper Lalonde (1986)-type studies estimate bias remaining after matching but generally can’t disentangle residual bias in POC from bias 2 Shadish & Clark paper solves this problem  Theoretically-informed matching can eliminate most/all of bias in POC  What about bias 2?  Extensions? can use this to assess average effects in population of those who would select the treatment if available

7. How well do quasi-experimental methods do at eliminating bias?  lessons of the Bloom paper consider ways of reducing variance of estimated treatment effect need to worry about functional form tie-breaking experiments enable us to evaluate bias in regression discontinuity (Black, Galdo, Smith 2005)  RD estimates sensitive to functional form unless cases are near threshold  treatment effect varies across thresholds

8. How well do quasi-experimental methods do at eliminating bias?  lessons of the Raudenbush paper adaptive centering provides same estimates as two-way fixed effects models better estimates of uncertainty, computationally easier, than fixed effects  under what conditions do such designs reduce bias? fixed-effects (or centering) eliminates bias in POC under the assumption the within-cell assignment to treatment is ignorable (under what conditions is this reasonable?) fixed-effects may increase bias 2 by reducing region of common support (domain of POC)

9. remaining questions  important to consider population of interest in research design, concerns about external validity  how can we asses the extent to which we should worry about bias 2? meta-analysis of multiple studies with different populations of interest? multi-site randomized trials draw study samples from known population, assess participation selection