1. Regression in Practice: Observational studies with controls for pretests work better than you think Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association, 103(484),
Quantify how much biased removed by statistical
control using pretests in a given setting
Sample: Volunteer undergraduates
Outcome: Math and vocabulary tests
Treatment:
basic didactic,
showing transparencies
defining math concepts
Berk R (2005) Randomized experiments as the bronze standard. J Exp Criminol 1:417–433
Berk R, Barnes G, Ahlman L, Kurtz E (2010) When second best is good enough: a comparison between a
true experiment and a regression discontinuity quasi-experiment. J Exp Criminol 6:191–208. ‘‘the results
from the two approaches are effectively identical’’ page 191.
Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009). Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis, 31(4),
Concato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, controlled trials, observational studies, and the hierarchy of research designs. New England Journal of Medicine, 342(25),
Cook, T. D., Shadish, S., & Wong, V. A. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy and Management. 27 (4), 724–750.
Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association, 103(484),
Steiner, Peter M., Thomas D. Cook & William R. Shadish (in press). On the importance of reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics.
Steiner, Peter M., Thomas D. Cook, William R. Shadish & M.H. Clark (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods. Volume 15, Issue 3. Pages more than just pretest.
Kane, T., & Staiger, D. (2008). Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation. NBER working paper
35. Kane, T., & Staiger, D. (2008).
Bifulco, Robert . "Can Nonexperimental Estimates Replicate Estimates Based on Random Assignment in Evaluations of School Choice? A Within‐Study Comparison." Journal of Policy Analysis and Management 31, no. 3 (2012):
Reports 64% to 96% reduced with pre-test
Guarino, C. M., Reckase, M. D., & Wooldridge, J. M. (2014). Can value-added measures of teacher performance be trusted?. Retrieved from
Advocates dynamic regression, which is essentially ER(1), control for prior
OLS Regression with pretests removes 84% to 94%
of bias relative to RCT!! Propensity by strata not quite as good

2. Supporting findings
Berk R (2005) Randomized experiments as the bronze standard. J Exp Criminol 1:417–433
Berk R, Barnes G, Ahlman L, Kurtz E (2010) When second best is good enough: a comparison between a true experiment and a regression discontinuity quasi-experiment. J Exp Criminol 6:191–208. ‘‘the results from the two approaches are effectively identical’’ page 191.
Bifulco, Robert . "Can Nonexperimental Estimates Replicate Estimates Based on Random Assignment in Evaluations of School Choice? A Within‐Study Comparison." Journal of Policy Analysis and Management 31, no. 3 (2012):
Chetty, Raj. John N. Friedman, and Jonah E. Rockoff. Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates. American Economic Review 2014, 104(9): 2593–2632
Concato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, controlled trials, observational studies, and the hierarchy of research designs. New England Journal of Medicine, 342(25),
Cook, T. D., P. Steiner, and S. Pohl  Assessing how bias reduction is influenced by covariate choice, unreliability and data analytic mode: An analysis of different kinds of within-study comparisons in different substantive domains. Multivariate Behavioral Research 44(6):  
Cook, T. D., Shadish, S., & Wong, V. A. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy and Management. 27 (4), 724–750.
Dong, N., & Lipsey, M. W. (2017). Can Propensity Score Analysis Approximate Randomized Experiments Using Pretest and Demographic Information in Pre-K Intervention Research?. Evaluation review, X (generally close using PSA, some differences in estimated effects)
Kane, T., & Staiger, D. (2008). Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation. NBER working paper
Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009). Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis, 31(4),
Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association, 103(484),
Steiner, Peter M., Thomas D. Cook & William R. Shadish (in press). On the importance of reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics.
Steiner, Peter M., Thomas D. Cook, William R. Shadish & M.H. Clark (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods. Volume 15, Issue 3. Pages
Wong, V. C., Valentine, J. C., & Miller-Bains, K. (2017). Empirical performance of covariates in education observational studies. Journal of Research on Educational Effectiveness, 10(1),
Pretests account for 80%-95% of the bias compared to RCT

3. Wong, V. C. , Valentine, J. C. , & Miller-Bains, K. (2017)
Wong, V. C., Valentine, J. C., & Miller-Bains, K. (2017). Empirical performance of covariates in education observational studies. Journal of Research on Educational Effectiveness, 10(1), see
Still, 17% of the bias remained for math and 28% for reading.
Here, inclusion of a single pretest measure resulted in 3% remaining bias for math, and 7% for reading
Thus, using the pretest measure alone, the remaining bias in the observational study ranged from 11% to 29% for the vocabulary outcome, and for math, it was more than double, ranging from 61% to 71%. Proxy pre-test used

4. Theory based covariates also helped!
Using multiple pre-tests
Wong, V. C., Valentine, J. C., & Miller-Bains, K. (2017). Empirical performance of covariates in education observational studies. Journal of Research on Educational Effectiveness, 10(1),
see
Theory based covariates also helped!

8. Cook, T. D. , P. Steiner, and S. Pohl. 2010
Cook, T. D., P. Steiner, and S. Pohl  Assessing how bias reduction is influenced by covariate choice, unreliability and data analytic mode: An analysis of different kinds of within-study comparisons in different substantive domains. Multivariate Behavioral Research 44(6):  

11. Reviews in Recent studies
Need:
rich covariates, local comparisons (within districts), multiple time points for pre-tests, multiple content areas for pre-tests
Chaplin, D., Mamun, A., Protik, A., Schurrer, J., Vohra, D., Bos, K., Burak, H., Meyer, L., Dumitrescu,
A., Ksoll, K., &Cook, T. (2016). Grid Electricity Expansion in Tanzania by MCC: Findings from a
Rigorous Impact Evaluation. Working Paper. Princeton, NJ: Mathematica Policy Research.
Hallberg, K., Cook, T. D., Steiner, P., & Clark, M. H. (in press[b]). The role of pretests in education
observational studies: Evidence from an empirical within study comparison. Prevention Science.
Hallberg, K., Wong, V., & Cook T. D. (in press[a]). Evaluating methods for selecting school level
comparisons in quasi-experimental designs: results from a within-study comparison. Journal of
Public Policy and Management.
St. Clair, T., Hallberg, K., & Cook, T. D. (in press). The validity and precision of the comparative
interrupted time-series design: three within-study comparisons. Journal of Educational and
Behavioral Statistics,
Wong, V.C., Valentine, J., Miller-Bain, K. (in press) Empirical Performance of Covariates in Education
Observational Studies. Journal of Research on Educational Effectiveness.

12. Criticisms of propensity scores
No better than the covariates that go into it
no control for unobservables
Heckman, 2005; Morgan & Harding, 2006, page 40; Rosenbaum, 2002, page 297; Shadish et al., 2002, page 164);
How could they be better than covariates?
Propensity=f(covariates).
Ambivalent about quality of propensity model
Group overlap must be substantial
Propensity model should not fit too well!
implies confounding of covariates and treatment
not good enough implies poorly understood treatment mechanism – poor control
Short-term biases (2 years) are substantially less than medium term (3 to 5 year) biases—the value of comparison groups may deteriorate
Heckman, J. (2005). The Scientific Model of Causality. Sociological Methodology, 35,
Morgan, S. L. & Harding, D. J. (2006). Matching estimators of causal effects: Prospects and pitfalls in theory and practice. Sociological Methods and Research 35, 3-60.
Rosenbaum, P Observational Studies. New York: Springer.
(Shadish, Cook, & Campbell, 2002)
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, NY: Houghton Mifflin.

13. Page 40: Morgan, S. L. & Harding, D. J. (2006)
Page 40: Morgan, S. L. & Harding, D. J. (2006). Matching estimators of causal effects: Prospects and pitfalls in theory and practice. Sociological Methods and Research 35, 3-60.
Page 101: Jenkins, J.M., Farkas, G., Duncan, G.J., Burchinal, M. and Vandell, D.L., Head Start at Ages 3 and 4 Versus Head Start Followed by State Pre-K Which Is More Effective?. Educational evaluation and policy analysis, 38(1), pp