1. Alternative Scenarios and Related Techniques
Non-zero null hypothesis Type II error: failure to reject the null when it is false Alternative Thresholds Related Techniques Confidence Intervals

2. Alternative Hypotheses: Non Zero Null Hypothesis
Non-zero null hypotheses (for kindergarten retention) H0:δ> −6. se x tcritical, df=7639=.68 x (−1.645)= −1.12 (one tailed test).

3. Alternative Hypotheses: Non Zero Null Hypothesis
δ# = −6−1.12=−7.12 1− δ #/ =1− (−7.12/−9)= % of estimated effect would have to be due to bias to invalidate inference for H0:δ> −6.

4. Alternative Scenario: Failure to Reject Null when null is False (type II error)
Assume the data are comprised of two subsamples, one with an effect at the threshold (r#) and the other (r ) with 0 effect, and define  as the proportion of the sample that has 0 effect.
rxy =(1-)r# +r xy  .
*Frank, K. A. and Min, K Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, * co first authors.

5. Sample Replacement: Estimate Does not Exceed the Threshold
Set rxy=0 and solve for :
=1- rxy /r#
If  of the cases have 0 effect then if you replace them (with cases at the threshold) then the overall estimate will be at the threshold.

6. Dehejia, Rajeev H. , and Sadek Wahba
Dehejia, Rajeev H., and Sadek Wahba. "Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs." Journal of the American statistical Association 94, no. 448 (1999):

8. Alternative Scenario: Alternative Thresholds for Inference
Use δ# = −4
To Invalidate the inference, 56% of the estimate would have to be due to bias if the threshold for inference is -4.

9. Alternative Scenario: Non-zero Effect in the Replacement (e. g
Alternative Scenario: Non-zero Effect in the Replacement (e.g., non-volunteer) Population
Inference invalid if 1-πp<(δp − δ#)/(δ p − δ p´ ). If δ p´ = −2, and δ#=3.68 and δ p =7.95 (both as in the initial example for Open Court). Inference is invalid if 1-πp<(7.95 – 3.68)/(7.95 − −2 ) =.43
inference invalid if more than 43% of the sample were replaced with cases for which the effect of OCR was −2 .

10. Related Techniques
Bounding (e.g., Altonji et, Elder & Tabor, 2005; Imbens 2003; Manski)
lower bound: “if unobserved factors are as strong as observed factors, how small could the estimate be?”
Focus on estimate
% robustness: “how strong would unobserved factors have to be to invalidate inference?”
Focus on inference, policy & behavior
External validity based on propensity to be in a study (Hedges and O’Muircheartaigh )
They focus on estimate
We focus on comparison with a threshold
Other sensitivity (e.g., Rosenbaum or Robins)
Characteristics of variables needed to change inference
We focus on how sample must change.
Can be applied to observational study or RCT
Other Sources of Bias
Violations of SUTVA
Agent based models?
Measurement error
Just another source of bias (minor concern for examples here)
Differential treatment effects
Use propensity scores to differentiate, then apply indices
Altonji, J.G., Elder, T., and Taber, C. (2005). An Evaluation of Instrumental Variable Strategies for Estimating the Effects of Catholic Schooling. Journal of Human Resources 40(4):
Altonji, J.G., Conley, T., Elder, T., and Christopher Taber. (2010). ``Methods for Using Selection on Observed Variables to Address Selection on Unobserved Variables’’. Retrieved from
Hedges, L. , O’Muircheartaigh, C. (2011). Generalization from Experiments. Retrieved from
Imbens, G “Sensitivity to Exogeneity Assumptions in Program Evaluation”American Economic Review, Papers and Proceedings, 93(2),
Rosenbaum, P. R., and Rubin, D. B. (1983). Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. Journal of the Royal Statistical Society (Series B), 45,

11. δ # statistical significance threshold for making an inference
Relationship between the Confidence Interval and % Bias Necessary to Invalidate
the Inference of an Effect of Open Court on Comprehensive Reading Score
Lower bound of confidence interval “far from 0” estimate exceeds threshold
by large amount
Confidence Interval
Closer to 0? } } }
δ #
δ #
δ # statistical significance threshold for making an inference
estimated effect

12. Exercise: Different Scenarios
Using an inference you have identified or one of the ones I presented, use the spreadsheet to evaluate:
Different thresholds
Alternative hypotheses
Type II errors
Non-zero effects in replacement cases