Comparison of Correlation and % Bias to Invalidate Frameworks Different bases for evaluating r-r# Regression as linear adjustment to replace cases Applications of the Frameworks

% bias necessary to invalidate the inference .9 .8 .7 .6 .5 .4 .3 .2 .1 r % bias necessary to invalidate the inference { r } r#

.9 .8 .7 .6 .5 .4 .3 .2 .1 r r# r r# Robustness refers to alternatve world with different sample or control variables. It’s a counterfactual world. MAYBE NEED A SLIDE THAT HAS SAME R BUT DIFFERENT THRESHOLD? R# scales by different n (assuming threshold defined by stat sig) versus % bias that scales by effect size. One scales by the study design (sample size) then other scales by the study result (effect size).

.9 .8 .7 .6 .5 .4 .3 .2 .1 r r r# r r# Robustness refers to alternatve world with different sample or control variables. It’s a counterfactual world. MAYBE NEED A SLIDE THAT HAS SAME R BUT DIFFERENT THRESHOLD? R# scales by different n (assuming threshold defined by stat sig) versus % bias that scales by effect size. ITCV gives credit to large effects exceeding large thresholds (for small n). The % bias gives credit to smaller effects exceeding smaller thresholds. If threshold base don stat sig, then % bias gives more credit to larger studies, ITCV gives more credit to smaller studies. ITCV scales by the study design (gives credt to smaller sample sizes), % bias scales by the study result (effect size).

.9 .8 .7 .6 .5 .4 .3 .2 .1 r r# r r# Robustness refers to alternatve world with different sample or control variables. It’s a counterfactual world.

Transforming % Bias to Replace to ITCV Hmmmm….

Reflection What part if most confusing to you? Why? More than one interpretation? Talk with one other, share Find new partner and problems and solutions

Linear Adjustment to Invalidate Inference Linear model distributes change over all cases How many cases would you have to replace with zero effect counterfactuals to change the inference? Assume threshold is 4 (δ# =4): 1- δ# / =1-4/6=.33 =(1/3) 9 9 10 11 6 3 4 5 +6=9 +3=6+2=6 +1=6 6.00 6 4 The inference would be invalid if you replaced 33% (or 1 case) with counterfactuals for which there was no treatment effect. New estimate=(1-% replaced) +%replaced(no effect)= (1-%replaced) =(1-.33)6=.66(6)=4

Control for Confound ↔Adjusting the Outcome Replacing observed cases with linear counterfactuals 6

Comparison of Frameworks % bias to invalidate Any estimate + standard error Case replacement Good for experimental settings (treatments) Think in terms of cases (people or schools) counterfactual or unsampled population Assume equal effect of replacing any case Or weighted cases with weighted replacements Correlational Good for continuous predictors Linear models only Think in terms of correlations, variables Both can be applied to Internal and External Validity any threshold