Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum What would it take to Change your Inference? Quantifying the Discourse about Causal Inferences in the Social Sciences Kenneth A. Frank Help from Yun-jia Lo and Mike Seltzer AERA workshop April 19, 2015 (AERA on-line video – cost is $95)AERA on-line video – cost is $95 Motivation Statistical inferences are often challenged because of uncontrolled bias. There may be bias due to uncontrolled confounding variables non-random selection into a sample. We will answer the question about what it would take to change an inference by formalizing the sources of bias and quantifying the discourse about causal inferences in terms of those sources. For example, we will transform challenges such as “But the inference of a treatment effect might not be valid because of pre-existing differences between the treatment groups” to questions such as “How much bias must there have been due to uncontrolled pre-existing differences to make the inference invalid?” Approaches In part I we will use Rubin’s causal model to interpret how much bias there must be to invalidate an inference in terms of replacing observed cases with counterfactual cases or cases from an unsampled population. In part II, we will quantify the robustness of causal inferences in terms of correlations associated with unobserved variables or in unsampled populations. Calculations for bivariate and multivariate analysis will be presented in the spreadsheet for calculating indices [KonFound-it!] with some links to SPSS, SAS, and Stata.spreadsheet for calculating indices [KonFound-it!] Format The format will be a mixture of presentation, individual exploration, and group work. Participants may include graduate students and professors, although all must be comfortable with basic regression and multiple regression. Participants should bring their own laptop, or be willing to work with another student who has a laptop. Participants may choose to bring to the course an example of an inference from a published study or their own work, as well as data analyses they are currently conducting. Materials to download: spreadsheet for calculating indices [KonFound-it!]spreadsheet for calculating indices [KonFound-it!] powerpoint with examples and calculations

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum overview Replacement Cases FrameworkReplacement Cases Framework (40 minutes to reflection) Thresholds for inference and % bias to invalidate and inference The counterfactual paradigm Application to concerns about non-random assignment to treatments Application to concerns about non-random sample Reflection (10 minutes)Reflection Examples of replacement framework Internal validity example: Effect of kindergarten retention on achievement (40 minutes to break) External validity example: effect of Open Court curriculum on achievement Review and Reflection Extensions Extensions of the framework Exercise and breakExercise and break (20 minutes: Yun-jia Lo, Mike Seltzer support) Correlational FrameworkCorrelational Framework (25 minutes to exercise) How regression works Impact of a Confounding variable Internal validity: Impact necessary to invalidate an inference Example: Effect of kindergarten retention on achievement ExerciseExercise (25 minutes: Mike Seltzer supports on-line) External validity (30 minutes) combining estimates from different populations example: effect of Open Court curriculum on achievement ConclusionConclusion (10 minutes)

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum https://www.msu.edu/~kenfrank/research.htm#causal Materials for course

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Quick Survey Can you make a causal inference from an observational study?

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Answer: Quantifying the Discourse Can you make a causal inference from an observational study? Of course you can. You just might be wrong. It’s causal inference, not determinism. But what would it take for the inference to be wrong?

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum I: Replacement of Cases Framework How much bias must there be to invalidate an inference? Concerns about Internal Validity What percentage of cases would you have to replace with counterfactual cases (with zero effect) to invalidate the inference? Concerns about External Validity What percentage of cases would you have to replace with cases from an unsampled population (with zero effect) to invalidate the inference?

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum What Would It Take to Change an Inference? Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences Abstract We contribute to debate about causal inferences in educational research in two ways. First, we quantify how much bias there must be in an estimate to invalidate an inference. Second, we utilize Rubin’s causal model (RCM) to interpret the bias necessary to invalidate an inference in terms of sample replacement. We apply our analysis to an inference of a positive effect of Open Court Curriculum on reading achievement from a randomized experiment, and an inference of a negative effect of kindergarten retention on reading achievement from an observational study. We consider details of our framework, and then discuss how our approach informs judgment of inference relative to study design. We conclude with implications for scientific discourse. Keywords: causal inference; Rubin’s causal model; sensitivity analysis; observational studies Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. 2013. What would it take to Change an Inference?: Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences. Education, Evaluation and Policy Analysis. Vol 35: 437-460. http://epa.sagepub.com/content/early/recentFrank, K.A., Maroulis, S., Duong, M., and Kelcey, B. 2013. What would it take to Change an Inference?: Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences. Education, Evaluation and Policy Analysis. Vol 35: 437-460. http://epa.sagepub.com/content/early/recent

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum { } % bias necessary to invalidate the inference

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Quantifying the Discourse: Formalizing Bias Necessary to Invalidate an Inference δ =a population effect, =the estimated effect, and δ # =the threshold for making an inference An inference is invalid if: (1) An inference is invalid if the estimate is greater than the threshold while the population value is less than the threshold. Defining bias as -δ, (1) implies an estimate is invalid if and only if: Expressed as a proportion of the estimate, inference invalid if:

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Algebra for % Bias to Invalidate

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum δ#δ# { } % bias necessary to invalidate the inference

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Interpretation of % Bias to Invalidate an Inference % Bias is intuitive Relates to how we think about statistical significance Better than “highly significant” or “barely significant” But need a framework for interpreting

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Framework for Interpreting % Bias to Invalidate an Inference: Rubin’s Causal Model and the Counterfactual 1)I have a headache 2)I take an aspirin (treatment) 3)My headache goes away (outcome) Q) Is it because I took the aspirin? A)We’ll never know – it is counterfactual – for the individual This is the Fundamental Problem of Causal Inference

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Definition of Replacement Cases as Counterfactual: Potential Outcomes Definition of treatment effect for individual i: Fundamental problem of causal inference is that we cannot simultaneously observe

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Fundamental Problem of Inference and Approximating the Counterfactual with Observed Data (Internal Validity) 345345 6? But how well does the observed data approximate the counterfactual? 9 10 11

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Symbolic: Fundamental Problem of Inference and Approximating the Counterfactual with Observed Data (Internal Validity) 6? But how well does the observed data approximate the counterfactual? Y t |X=t Y c |X=t Y c |X=c Y t |X=c

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Approximating the Counterfactual with Observed Data 345345 But how well does the observed data approximate the counterfactual? Difference between counterfactual values and observed values for the control implies the treatment effect of 1 8 9 10 111111 6 is overestimated as 6 using observed control cases with mean of 4 9

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Using the Counterfactual to Interpret % Bias to Invalidate the Inference How many cases would you have to replace with zero effect counterfactuals to change the inference? Assume threshold is 4 (δ # =4): 1- δ # / =1-4/6=.33 =(1/3) 666666 6.00 The inference would be invalid if you replaced 33% (or 1 case) with counterfactuals for which there was no treatment effect. New estimate=(1-% replaced) +%replaced(no effect)= (1-%replaced) =(1-.33)6=.66(6)=4 000000 6 4 345345 10 11 9

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Which Cases to Replace Think of it as an expectation: if you randomly replaced 1 case, and repeated 1,000 times, on average the new estimate would be 4 Assumes constant treatment effect conditioning on covariates and interactions already in the model

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum δ#δ# { } % bias necessary to invalidate the inference To invalidate the inference, replace 33% of cases with counterfactual data with zero effect

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Fundamental Problem of Inference and Approximating an Unsampled Population with Observed Data (External Validity) 9 10 Y t |Z=p 11 3 4 Y c |Z=p 5 64 How many cases from p would you have to replace with cases with zero effect from p` to change the inference? Assume threshold is: δ # =4: 1- δ # / =1-4/6=.33 =(1/3) 6 Y t |Z=p´ 6 Y c |Z=p´ 0 66 6 666 6 6

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum δ#δ# { } % bias necessary to invalidate the inference To invalidate the inference, replace 33% of cases with cases from unsampled population data with zero effect

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Review & Reflection Review of Framework Pragmatism  thresholds How much does an estimate exceed the threshold  % bias to invalidate the inference Interpretation: Rubin’s causal model internal validity: % bias to invalidate  number of cases that must be replaced with counterfactual cases (for which there is no effect) external validity: % bias to invalidate  number of cases in sample population p that must be replaced with unobserved population p` (for which there is no effect) Reflect Which part is most confusing to you? Is there more than one interpretation? Discuss with a partner or two

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Example of Internal Validity from Observational Study : The Effect of Kindergarten Retention on Reading and Math Achievement (Hong and Raudenbush 2005) 1. What is the average effect of kindergarten retention policy? (Example used here) Should we expect to see a change in children’s average learning outcomes if a school changes its retention policy? Propensity based questions (not explored here) 2. What is the average impact of a school’s retention policy on children who would be promoted if the policy were adopted? Use principal stratification. Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics. Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205–224

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Data Early Childhood Longitudinal Study Kindergarten cohort (ECLSK) US National Center for Education Statistics (NCES). Nationally representative Kindergarten and 1 st grade observed Fall 1998, Spring 1998, Spring 1999 Student background and educational experiences Math and reading achievement (dependent variable) experience in class Parenting information and style Teacher assessment of student School conditions Analytic sample (1,080 schools that do retain some children) 471 kindergarten retainees 10,255 promoted students

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Effect of Retention on Reading Scores (Hong and Raudenbush)

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Possible Confounding Variables (note they controlled for these) Gender Two Parent Household Poverty Mother’s level of Education (especially relevant for reading achievement) Extensive pretests measured in the Spring of 1999 (at the beginning of the second year of school) standardized measures of reading ability, math ability, and general knowledge; indirect assessments of literature, math and general knowledge that include aspects of a child’s process as well as product; teacher’s rating of the child’s skills in language, math, and science

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Calculating the % Bias to Invalidate the Inference: Obtain spreadsheet From https://www.msu.edu/~kenfrank/research.htm#causalhttps://www.msu.edu/~kenfrank/research.htm#causal Choose spreadsheet for calculating indices Access spreadsheet spreadsheet for calculating indices [KonFound-it!]

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Kon-Found-it: Basics

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Obtain t critical, estimated effect and standard error Estimated effect ( ) = -9.01 Standard error=.68 n=7168+471=7639; df > 500, t critical =-1.96 From: Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics. Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205–224

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum covariates Page 215 Df=207+14+2=223

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Kon-Found-it: Basics % Bias to Invalidate Estimated effect ( ) = -9.01 Standard error=.68 n=7168+471=7639 Covariates=223

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Calculating the % Bias to Invalidate the Inference: Inside the Calculations δ # =the threshold for making an inference = se x t critical, df>230 =.68 x -1.96=-1.33 [user can specify alternative threshold] % Bias necessary to invalidate inference = 1-δ # / =1-1.33/-9.01=85% 85% of the estimate must be due to bias to invalidate the inference. }

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Using the Counterfactual to Interpret % Bias to Invalidate the Inference How many cases would you have to replace with zero effect counterfactuals to change the inference? Assume threshold is 4 (δ # =4): 1- δ # / =1-4/6=.33 =(1/3) 666666 6.00 The inference would be invalid if you replaced 33% (or 1 case) with counterfactuals for which there was no treatment effect. New estimate=(1-% replaced) +%replaced(no effect)= (1-%replaced) =(1-.33)6=.66(6)=4 000000 6 4 345345 10 11 9

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Original cases that were not replaced Replacement counterfactual cases with zero effect Original distribution RetainedPromoted Example Replacement of Cases with Counterfactual Data to Invalidate Inference of an Effect of Kindergarten Retention Counterfactual: No effect

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Interpretation 1) Consider test scores of a set of children who were retained that are considerably lower (9 points) than others who were candidates for retention but who were in fact promoted. No doubt some of the difference is due to advantages the comparable others had before being promoted. But now to believe that retention did not have an effect one must believe that 85% of those comparable others would have enjoyed most (7.2) of their advantages whether or not they had been retained. This is even after controlling for differences on pretests, mother’s education, etc. 2) The replacement cases would come from the counterfactual condition for the observed outcomes. That is, 85% of the observed potential outcomes must be unexchangeable with the unobserved counterfactual potential outcomes such that it is necessary to replace those 85% with the counterfactual potential outcomes to make an inference in this sample. Note that this replacement must occur even after observed cases have been conditioned on background characteristics, school membership, and pretests used to define comparable groups. 3) Could 85% of the children have manifest an effect because of unadjusted differences (even after controlling for prior achievement, motivation and background) rather than retention itself?

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Evaluation of % Bias Necessary to Invalidate Inference Compare Bias Necessary to Invalidate Inference with Bias Accounted for by Background Characteristics 1% of estimated effect accounted for by background characteristics (including mother’s education), once controlling for pretests More than 85 times more unmeasured bias necessary to invalidate the inference Compare with % Bias necessary to invalidate inference in other studies Use correlation metric Adjusts for differences in scale

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum % Bias Necessary to Invalidate Inference based on Correlation to Compare across Studies t taken from HLM: =-9.01/.68=-13.25 n is the sample size q is the number of parameters estimated Where t is critical value for df>200 % bias to invalidate inference=1-.023/.152=85% Accounts for changes in regression coefficient and standard error Because t(r)=t(β)

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum % Bias Necessary to Invalidate Inference based on Correlation to Compare across Studies % bias to invalidate inference=1-.023/.155=85%

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Compare with Bias other Observational Studies

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum % Bias to Invalidate Inference for observational studies on-line EEPA July 24-Nov 15 2012 Kindergarten retention effect

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Exercise 1 : % Bias necessary to Invalidate an Inference Take an example from an observational study in your own data or an article Calculate the % bias necessary to invalidate the inference Interpret the % bias in terms of sample replacement What are the possible sources of bias? Would they all work in the same direction? Debate your inference with a partner

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum 44 Application to Randomized Experiment: Effect of Open Court Curriculum on Reading Achievement Open Court “scripted” curriculum versus business as usual 917 elementary students in 49 classrooms Comparisons within grade and school Outcome Measure: Terra Nova comprehensive reading score Borman, G. D., Dowling, N. M., and Schneck, C. (2008). A multi-site cluster randomized field trial of Open Court Reading. Educational Evaluation and Policy Analysis, 30(4), 389-407.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Value of Randomization Few differences between groups But done at classroom level Teachers might talk to each other School level is expensive (Slavin, 2008) Slavin, R. E. (2008). Perspectives on evidence-based research in education-what works? Issues in synthesizing educational program evaluations. Educational Researcher, 37, 5-14.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum n=27+22=49

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Anadjusted difference of 10

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Obtaining # parameters estimated, t critical, estimated effect and standard error Estimated effect ( ) = 7.95 Standard error=1.83 3 parameters estimated, Df=n of classrooms- # of parameters estimated= 49-3=46. t critical = t.05, df=46 =2.014

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Quantifying the Discourse for Borman et al: What would it take to change the inference? δ = population effect, =the estimated effect = 7.95, and δ # =the threshold for making an inference = se x t critical, df=46 =1.83 x 2.014=3.69 % Bias necessary to invalidate inference = 1- δ # / =1-3.69/7.95=54% 54% of the estimate must be due to bias to invalidate the inference

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Calculating the % Bias to Invalidate the Inference: Inside the Calculations =the estimated effect = 7.95 standard error =1.83 t critical= 2.014 % Bias necessary to invalidate inference = 1-d # /d =1-3.68/7.95=54%

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Calculating the % Bias to Invalidate the Inference: Inside the Calculations δ # =the threshold for making an inference = se x t critical, df=46 =1.83 x 2.014=3.686[user can override to specify hreshold] % Bias necessary to invalidate inference = 1-d # /d =1-3.686/7.95=54%

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum % Exceeding Threshold for Open Court Estimated Effect δ # =3.68 54 % above threshold=1-3.68/7.95=.54 } 54% of the estimate must be due to bias to invalidate the inference

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Fundamental Problem of Inference and Approximating the Counterfactual with Observed Data (External Validity) 9 10 11 3 4 5 66 6 666 6 6 64 How many cases would you have to replace with cases with zero effect to change the inference? Assume threshold is: δ # =4: 1- δ # / =1-4/6=.33 =(1/3) 6 6 0

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Interpretation of Amount of Bias Necessary to Invalidate the Inference: Sample Representativeness To invalidate the inference: 54% of the estimate must be due to sampling bias to invalidate Borman et al.’s inference You would have to replace 54% of Borman’s cases (about 30 classes) with classes in which Open Court had no effect to invalidate the inference Are 54% of Borman et al.’s cases irrelevant for non- volunteer schools? We have quantified the discourse about the concern of validity

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Example Replacement of Cases from Non-Volunteer Schools to Invalidate Inference of an Effect of the Open Court Curriculum Open Court Business as Usual Original volunteer cases that were not replaced Replacement cases from non-volunteer schools with no treatment effect Original distribution for all volunteer cases Business as Usual Unsampled Population: No effect

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum The Fundamental Problem of External Validity Before a randomized experiment: People believe they do not “know” what generally works People choose treatments based on idiosyncratic conditions -- what they believe will work for them (Heckman, Urzua and Vytlacil, 2006) After a randomized experiment: People believe they know what generally works People are more inclined to choose a treatment shown to generally work in a study because they believe “it works” The population is fundamentally changed by the experimenter (Ben- David; Kuhn) The fundamental problem of external validity the more influential a study the more different the pre and post populations, the less the results apply to the post experimental population All the more so if it is due to the design (Burtless, 1995)

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Restatement of the Fundamental Problem of External Validity A researcher cannot simultaneously change the behavior in a population and claim that the pre-study population fully represents the post- study population.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology. Journal of Quantitative Criminology, 26(4), 489-500. page 495

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology. Journal of Quantitative Criminology, 26(4), 489-500. page 496

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Comparisons across Randomized Experiments (correlation metric)

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Distribution of % Bias to Invalidate Inference for Randomized Studies EEPA: On-line Jul 24-Nov 5 2012

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Review & Reflection Review of applications Concern about internal validity: Kindergarten retention (Hong and Raudenbush) 85% of cases must be replaced counterfactual data (with no effect) to invalidate the inference of a negative effect of retention on reading achievement –Comparison with other observational studies Concern about external validity: Open Court Curriculum 54% of cases must be replaced with data from unobserved population to invalidate the inference of a positive effect of Open Court on reading achievement in non-volunteer schools –Comparison with other randomized experiments Reflect Which part is most confusing to you? Is there more than one interpretation? Discuss with a partner or two

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Exercise 2 : % Bias necessary to Invalidate an Inference Take an example of a randomized experiment in your own data or an article Calculate the % bias necessary to invalidate the inference Interpret the % bias in terms of sample replacement What are the possible sources of bias? Would they all work in the same direction? Debate your inference with a new partner

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Extensions of the Framework Ordered thresholds for decision- making Alternative hypotheses and scenarios Relationship to confidence intervals Related techniques

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Ordered Thresholds Relative to Transaction Costs 1.Changing beliefs, without a corresponding change in action. 2.Changing action for an individual (or family) 3.Increasing investments in an existing program. 4.Initial investment in a pilot program where none exists. 5.Dismantling an existing program and replacing it with a new program. Definition of threshold: the point at which evidence from a study would make one indifferent to policy choices

How District Leaders Really Make Decisions, and About What from Design-Based Implementation Research: A Strategic Approach to Scaling and Sustaining Educational Innovation: William R. Penuel University of Colorado Philip Bell University of Washington

Sampson, R. J. (2010). Gold standard myths: Observations on the experimental turn in quantitative criminology. Journal of Quantitative Criminology, 26(4), 489-500.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Alternative Hypotheses: Non Zero Null Hypothesis Non-zero null hypotheses (for kindergarten retention) H 0 :δ> −6. se x t critical, df=7639 =.68 x (−1.645)= −1.12 (one tailed test).

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Alternative Hypotheses: Non Zero Null Hypothesis δ # = −6−1.12=−7.12 1− δ # / =1− (−7.12/−9)=.21. 21% of estimated effect would have to be due to bias to invalidate inference for H 0 :δ> −6.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Alternative Scenario: Failure to Reject Null when null is False Assume the data are comprised of two subsamples, one with an effect at the threshold (d # ) and the other ( d ) with 0 effect, and define  as the proportion of the sample that has 0 effect. d xy =(1-  )d # +  d xy. *Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, 349-392. * co first authors.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Sample Replacement: Estimate Does not Exceed the Threshold Set d xy =0 and solve for  :  =1- d xy /d # =1-.5/1.183=.577=58% If  of the cases have 0 effect then if you replace them (with cases at the threshold) then the overall estimate will be at the threshold.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Dehejia, Rajeev H., and Sadek Wahba. "Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs." Journal of the American statistical Association 94, no. 448 (1999): 1053-1062.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Alternative Scenario: Alternative Thresholds for Inference Use δ # = −4 To Invalidate Hong and Raudenbush’s inference, 56% of the estimate would have to be due to bias if the threshold for inference is -4.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Alternative Scenario: Non-zero Effect in the Replacement (e.g., non-volunteer) Population Inference invalid if 1-π p <(δ p − δ # )/(δ p − δ p ´ ). If δ p ´ = −2, and δ # =3.68 and δ p =7.95 (both as in the initial example for Open Court). Inference is invalid if 1-π p <(7.95 – 3.68)/(7.95 − −2 ) =.43 inference invalid if more than 43% of the sample were replaced with cases for which the effect of OCR was −2.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum } 0 1 2 3 4 5 6 7 8 9 1 1 1 1 0 1 2 3 Confidence Interval Relationship between the Confidence Interval and % Bias Necessary to Invalidate the Inference of an Effect of Open Court on Comprehensive Reading Score δ # Lower bound of confidence interval “far from 0” estimate exceeds threshold by large amount 0 1 2 3 4 5 6 7 8 9 1 1 1 1 0 1 2 3 } } δ # } }

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Related Techniques Bounding (e.g., Altonji et, Elder & Tabor, 2005; Imbens 2003; Manski) lower bound: “if unobserved factors are as strong as observed factors, how small could the estimate be?” Focus on estimate % robustness: “how strong would unobserved factors have to be to invalidate inference?” Focus on inference, policy & behavior External validity based on propensity to be in a study (Hedges and O’Muircheartaigh ) They focus on estimate We focus on comparison with a threshold Other sensitivity (e.g., Rosenbaum or Robins) Characteristics of variables needed to change inference We focus on how sample must change. Can be applied to observational study or RCT Other Sources of Bias Violations of SUTVA Agent based models? Measurement error Just another source of bias (minor concern for examples here) Differential treatment effects Use propensity scores to differentiate, then apply indices

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum II: Correlation Framework Concerns about internal validity At what levels must an omitted variable be correlated with a predictor and outcome to change an inference about the predictor? Frank, K. 2000. "Impact of a Confounding Variable on the Inference of a Regression Coefficient." Sociological Methods and Research, 29(2), 147- 194 Concerns about external validity What must be the correlation between predictor and outcome in the unsampled population to invalidate an inference about an effect in a population that includes the unsampled population? *Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, 349-392. * co first authors. Combined application (with propensity scores) Frank, K.A., Gary Sykes, Dorothea Anagnostopoulos, Marisa Cannata, Linda Chard, Ann Krause, Raven McCrory. 2008. Extended Influence: National Board Certified Teachers as Help Providers. Education, Evaluation, and Policy Analysis. Vol 30(1): 3-30.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Making an Inference from an Observational Study 83 Gather data on people as they naturally choose “treatments” Measure covariates, especially pretests Randomly sample from a larger population – can do because not assigning to treatments Examples Coleman et al., Catholic schools effect Hong and Raudenbush: effects of kindergarten retention

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Linear Adjustment to Invalidate Inference How many cases would you have to replace with zero effect counterfactuals to change the inference? Assume threshold is 4 (δ # =4): 1- δ # / =1-4/6=.33 =(1/3) 666666 6.00 The inference would be invalid if you replaced 33% (or 1 case) with counterfactuals for which there was no treatment effect. New estimate=(1-% replaced) +%replaced(no effect)= (1-%replaced) =(1-.33)6=.66(6)=4 000000 6 4 345345 9 10 11 9 +3=6 +2=6 +1=6 +6=9 Linear model distributes change over all cases

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Adjusting for Confound to get Correct Effect 666666 To replace observed control cases to get estimated effect of 4 instead of uncontrolled mean difference of 6, use confound: control group disadvantaged from the start Replacing observed cases with linear counterfactuals But how do we obtain these estimates?

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Control for Confound ↔ Adjusting the Outcome 666666 Replacing observed cases with linear counterfactuals

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Regression! SAS Syntax for reading in toy counterfactual data DATA one; input y s confound; cards; 9 1 0 10 1 0 11 1 0 3 0 -3 4 0 -2 5 0 -1 run; proc reg data=one; model y=s ; run; proc reg data=one; model y=s confound; run; Biased estimated Unbiased estimate

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Regression!: SPSS Syntax for reading in toy counterfactual data DATA LIST FREE / y s confound. Begin DATA. 9 1 0 10 1 0 11 1 0 3 0 -3 4 0 -2 5 0 -1 End DATA. REGRESSION /MISSING LISTWISE /STATISTICS COEFF OUTS R ANOVA /CRITERIA=PIN(.05) POUT(.10) /NOORIGIN /DEPENDENT y /METHOD=ENTER s. REGRESSION /MISSING LISTWISE /STATISTICS COEFF OUTS R ANOVA /CRITERIA=PIN(.05) POUT(.10) /NOORIGIN /DEPENDENT y /METHOD=ENTER s confound. Biased estimated Unbiased estimate

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum How Regression Works: Partial Correlation Partial Correlation: correlation between s and y, where s and y have been controlled for the confounding variable Spss syntax CORRELATIONS /VARIABLES=y s confound /PRINT=TWOTAIL NOSIG Sas syntaxt proc corr data=one; var y s confound; run;

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Derivation of Partial from Matrix Algebra (Normal Equations) Multivariate: Β=[X’X] -1 X’Y For 2 variables that are standardized:

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Defining the Impact of a Confounding Variable

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum r sy|cv =.866 How Regression Works: Impact of a Confounding Variable on a Regression Coefficient r s∙y =.964 r cv∙y =.928.866x.928 s y r cv∙s =.866 r c∙s xr c∙y cv Impact weights the relationship between cv and y by the relationship between c and s: the stronger the relationship between cv and y, the more important the relationship between cv and s.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Alternative Conceptualization (overlapping variance) (r x2∙y|x1 ) 2

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Everything old is new again

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Regression works… As long as you have the right confounding variables Linear relationships assumed Does this work in practice?

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Regression in Practice: Observational studies with controls for pretests work better than you think Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association, 103(484), 1334-1344. Quantify how much biased removed by statistical control using pretests in a given setting Sample: Volunteer undergraduates Outcome: Math and vocabulary tests Treatment: basic didactic, showing transparencies defining math concepts OLS Regression with pretests removes 84% to 94% of bias relative to RCT!! Propensity by strata not quite as good See also Concato et al., 2000 for a comparable example in medical research

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Supporting findings Berk R (2005) Randomized experiments as the bronze standard. J Exp Criminol 1:417–433 Berk R, Barnes G, Ahlman L, Kurtz E (2010) When second best is good enough: a comparison between a true experiment and a regression discontinuity quasi-experiment. J Exp Criminol 6:191–208. ‘‘the results from the two approaches are effectively identical’’ page 191. Pohl, S., Steiner, P. M., Eisermann, J., Soellner, R., & Cook, T. D. (2009). Unbiased causal inference from an observational study: Results of a within-study comparison. Educational Evaluation and Policy Analysis, 31(4), 463-479. Concato, JConcato, J., Shah, N., & Horwitz, R. I. (2000). Randomized, controlled trials, observational studies, and the hierarchy of research designs. New England Journal of Medicine, 342(25), 1887-1889.Shah, NHorwitz, R. I Cook, T. D., Shadish, S., & Wong, V. A. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy and Management. 27 (4), 724– 750. Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association, 103(484), 1334-1344. Steiner, Peter M., Thomas D. Cook & William R. Shadish (in press). On the importance of reliable covariate measurement in selection bias adjustments using propensity scores. Journal of Educational and Behavioral Statistics. Steiner, Peter M., Thomas D. Cook, William R. Shadish & M.H. Clark (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods. Volume 15, Issue 3. Pages 250-267. more than just pretest.Volume 15, Issue 3 Kane, T., & Staiger, D. (2008). Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation. NBER working paper 14607. 35. Kane, T., & Staiger, D. (2008). Bifulco, Robert. "Can Nonexperimental Estimates Replicate Estimates Based on Random Assignment in Evaluations of School Choice? A Within ‐ Study Comparison." Journal of Policy Analysis and Management 31, no. 3 (2012): 729-751. Reports 64% to 96% reduced with pre-test

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum New paper on Multiple Approaches (from Tom Dietz) http://arxiv.org/pdf/1412.3773v1.pdf

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Reflection What part if most confusing to you? Why? More than one interpretation? Talk with one other, share Find new partner and problems and solutions

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum When Regression Doesn’t Work Instrumental variables? Alternative assumptions, may be not better –Exclusion restriction –Strong instrument –Large n Propensity scores? No better than the covariates that go into it Heckman, 2005; Morgan & Harding, 2006, page 40; Rosenbaum, 2002, page 297; Shadish et al., 2002, page 164); How could they be better than covariates? –Propensity=f(covariates).Propensity=f(covariates). Paul Holland says: PAUL ROSENBAUM

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Sensitivity Analysis: What Must be the Impact of an Unmeasured Confounding variable invalidate the Inference? Step 1: Establish Correlation Between predictor of interest and outcome Step 2: Define a Threshold for Inference Step 3: Calculate the Threshold for the Impact Necessary to Invalidate the Inference Step 4: Multivariate Extension, with other Covariates

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Obtain t critical, estimated effect and standard error Estimated effect ( ) = -9.01 Standard error=.68 n=7168+471=7639; df > 500, t critical =-1.96 From: Hong, G. and Raudenbush, S. (2005). Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics. Educational Evaluation and Policy Analysis. Vol. 27, No. 3, pp. 205–224

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Establish Correlation and Threshold for Kindergarten Retention on Achievement t taken from HLM: =-9.01/.68=-13.25 n is the sample size =7639 q is the number of parameters estimated ( predictor+omitted variable=2) Where t is critical value for df>200 Step 1: calculate treatment effect as correlation Step 2: calculate threshold for inference (e.g., statistical significance) r # can also be defined in terms of effect sizes

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Step 3a: Calculate the Threshold for the Impact Necessary to Invalidate the Inference Set r x∙y|cv =r # and solve for k to find the threshold for the impact of a confounding variable (TICV). Sometimes I say ITCV. Assume r x∙cv =r y∙cv (which maximizes the impact of the confounding variable – Frank, 2000). Then impact= r x∙cv x r y∙cv = r x∙cv x r x∙cv = r y∙cv x r y∙cv, and Magnitude of impact of an unmeasured confound >.130 → inference invalid Magnitude of impact of an unmeasured confound <.130 → inference valid.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Maximizing impact

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Step 3b: Component Correlations The magnitude of each correlation (r x∙cv, r y∙cv ) must be greater than.36 to change inference. The correlations must have opposite signs. If r x∙cv = r y∙cv =r, then impact= r x∙cv x r y∙cv =r 2. Component correlations for threshold

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Calculating the % Bias to Invalidate the Inference: Obtain spreadsheet From https://www.msu.edu/~kenfrank/research.htm#causalhttps://www.msu.edu/~kenfrank/research.htm#causal Choose spreadsheet for calculating indices Access spreadsheet spreadsheet for calculating indices [KonFound-it!]

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum KonFound-it! Spreadsheet User enters values in yellow

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Inside the Calculations

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Inside the Calculations Step 1: calculate treatment effect as correlation Step 2: calculate threshold for inference (e.g., statistical significance)

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Inside the Calculations: Step 3a Step 3b

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Inside the Calculations: Multivariate Step 1M: calculate treatment effect as correlation with DF correction Step 2M: calculate threshold for inference (e.g., statistical significance) DF correction

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Step 3M: Multivariate Extension, with Covariates k=r x ∙cv|z × r y ∙ cv|z Following bivariate approach: maximizing the impact with covariates z in the model implies And components

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Assumption: Omitted Confound Independent of Observed Covariates Gives maximal credence to skeptic challenging inference

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Step 3M: Multivariate Extension, with Covariates Following bivariate approach: maximizing the impact with covariates z in the model implies And

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Obtaining R 2 YZ :

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Inside Calculations: Obtaining R 2 YZ :

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Inside Calculations: Obtaining R 2 xz xZ :

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Extra Sufficient Statistics for Obtaining R 2 xZ number of covariates % retained= 471/7639=.061 Var(x)=.061*.94=.057, std(x)=.24 Page 208 Page 210 Residual variance=.55+.88=143 Initial variance=13.53 2 =183 R 2 =1-143/183=.22 or page 216: page 215: std(y), page 217 Already have Se(β 1 )

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Multivariate Calculations in Spreadsheet User enters Std(y) Std(x) R 2

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Obtaining R 2 xz :

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum What must be the Impact of an Unmeasured Confound to Invalidate the Inference? If |k| >.132 (or.130 without covariates) then the inference is invalid If |r x cv|z |= |r y cv|z |, then each would have to be greater than k 1/2 =.36 to invalidate the inference. *Because effect is negative, components take opposite signs *Correlations must be partialled for covariates z. Multivariate adjustment, to invalidate the inference: |r y cv | >.325 and |r x cv | >.310

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Links to sas and stata code sas program for calculating indices and related measures stata code for calculating indices (beta version – thanks to Yun-jia Lo)

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Alternative: Regression Coefficient and Standard Error

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Maximizing Expression

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Comparing Regression vs Correlation ITCV (example from Frank, 2000) Regression (page 155) Correlation (pages 160,182; Frank, Sykes et al., 2008)

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Exercise 3: Impact Threshold 1)Identify a statistical inference in an article you are interested in. 2) Describe possible confounds/alternative explanations that could bias the estimate 3) Note the sample size and t-ratio recall t=estimate/[se(estimate)] 4) Calculate robustness of inference using http://www.msu.edu/~kenfrank/papers/calculating%20indic es%203.xls 5) If you have the data calculate the multivariate ITCV 6) Discuss with a partner

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Applications of Impact Threshold in Education and Sociology (also some in accounting) Frank, 2000. ITCV=.228. Frank, K.A., Zhao, Y., Penuel, W.R., Ellefson, N.C., and Porter, S. 2011. Focus, Fiddle and Friends: Sources of Knowledge to Perform the Complex Task of Teaching. Sociology of Education, Vol 84(2): 137-156. Frank, K.A., Gary Sykes, Dorothea Anagnostopoulos, Marisa Cannata, Linda Chard, Ann Krause, Raven McCrory. 2008. Extended Influence: National Board Certified Teachers as Help Providers. Education, Evaluation, and Policy Analysis. Vol 30(1): 3-30. Frisco, Michelle, Muller, C. and Frank, K.A. 2007. Using propensity scores to study changing family structure and academic achievement. Journal of Marriage and Family. Vol 69(3): 721–741Journal of Marriage and Family *Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, 349-392. * co first authors. Frank, K. 2000. "Impact of a Confounding Variable on the Inference of a Regression Coefficient." Sociological Methods and Research, 29(2), 147-194 Crosnoe, Robert and Carey E. Cooper. 2010. “Economically Disadvantaged Children’s Transitions into Elementary School: Linking Family Processes, School Contexts, and Educational Policy.” American Educational Research Journal 47: 258-291. ITCV =.32 Crosnoe, Robert. 2009. “Low-Income Students and the Socioeconomic Composition of Public High Schools.” American Sociological Review 74: 709-730. ITCV=.03 Augustine, Jennifer March, Shannon Cavanagh, and Robert Crosnoe. 2009. “Maternal Education, Early Child Care, and the Reproduction of Advantage.” Social Forces 88: 1-30. ITCV=.4,.23 Maroulis, S. & Gomez, L. (2008). “Does ‘Connectedness’ Matter? Evidence from a Social Network Analysis within a Small School Reform.” Teachers College Record, Vol. 110, Issue 9. Cheng, Simon, Regina E. Werum, and Leslie Martin. 2007. “Adult Social Capital: How Family and Community Ties Shape Track Placement of Ethnic Groups in Germany.” American Journal of Education 114: 41-74. William Carbonaro1William Carbonaro1 Elizabeth Covay1 School Sector and Student Achievement in the Era of Standards Based Reforms. Sociology of eductaion vol. 83 no. 2 160-182.Elizabeth Covay1 Chen, Xiaodong, Frank, Kenneth A. Dietz, Thomas, and Jianguo Liu. 2012. Weak Ties, Labor Migration, and Environmental Impacts: Toward a Sociology of Sustainability. Organization & Environment. Vol. 25 no. 1 3-24. see also Pan, W., and Frank, K.A. (2004). "An Approximation to the Distribution of the Product of Two Dependent Correlation Coefficients." Journal of Statistical Computation and Simulation, 74, 419-443 Pan, W., and Frank, K.A., 2004. "A probability index of the robustness of a causal inference," Journal of Educational and Behavioral Statistics, 28, 315-337.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Extensions Logistic Simulation: Kelcey, B. 2009. Improving and Assessing Propensity Score Based Causal Inferences in Multilevel and Nonlinear Settings. Unpublished doctoral dissertation, University of Michigan Ichino, Andrea, Fabrizia Mealli, and Tommaso Nannicini. 2008. “From temporary help jobs to permanent employment: what can we learn from matching estimators and their sensitivity? ” Journal of Applied Econometrics 23:305–327. Nannicini, Tommaso. 2007. “Simulation–based sensitivity analysis for matching estimators.” Stata Journal 7:334–350. Table David J. Harding. 2003. “Counterfactual Models of Neighborhood Effects: The Effect of Neighborhood Poverty on High School Dropout and Teenage Pregnancy.” American Journal of Sociology 109(3): 676-719.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Extensions: Multilevel If omitted confound is at level 1, no concerns. Covariates include level 2 fixed or random effects If omitted confound at level 2: Seltzer, M. H., Frank, K. A., & Kim, J. (2007). Studying the Sensitivity of Inferences to Possible Unmeasured Confounding Variables in Multisite Evaluations. Paper presented at invited session at AERA 2007.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Extensions: Impact Not Maximized and Impact Curve Impacts above the blue line invalidate the inference Smallest impact to invalidate inference: r xcv =ry cv =.364

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Extensions: Impacts Before and After Pretests If k >.109 (or.130 without covariates) then the inference is invalid Impact of strongest measured covariate (student approaches to learning) is -.00126 (sign indicates controlling for it reduces estimated effect of retention)  Impact of unmeasured confound would have to be about 100 times greater than the impact of the strongest observed covariate to invalidate the inference. Hmmm….

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Effect of Retention on Achievement After Adding each Covariate ControlsEstSet School-21.24.63-33.49 School+Pre2 (spring Kindergarten)-12.01.45-26.48 School+Pre2+(Pre2-Pre1)+-12.10.47-26.28 School+Pre2+(Pre2-Pre1)+ Momed -12.00.47-26.26 School+Pre2+(Pre2-Pre1)+ Female -12.07.46-25.18 School+Pre2+(Pre2-Pre1)+ 2parent -12.01.46-26.27 School+Pre2+(Pre2-Pre1)+ poverty -12.04.46-26.16 Hong and Raudenbush (model based)-9.01.68-13.27 n=10,065, R 2 =.40 Note: 1 year’s growth is about 10 points, so retention effect > 1 year growth

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Consider Alternate Sample (External Validity) Causal Inference concern: We cannot assert cause if the effect of is not constant across contexts. Statistical Translation: Would the inference be valid if the sample included more of some population (e.g. non- volunteer schools) for which the effect was not as strong? Rephrased for robustness: what must be the conditions in the alternative sample to invalidate the inference?

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Consider Alternate Sample (External Validity) Define  as the proportion of the sample that is replaced with an alternate sample. r is correlation in unobserved data R is combined correlation for observed and unobserved data: R xy =(1-  )r xy +  r xy. *Frank, K. A. and Min, K. 2007. Indices of Robustness for Sample Representation. Sociological Methodology. Vol 37, 349-392. * co first authors.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Thresholds for Sample Replacement Set R =r # and solve for r xy : If half the sample is replaced (  =.5), original inference is invalid if r xy < 2r # -r xy Therefore, 2r # -r xy defines the threshold for replacement: TR(  =.5 ) If r xy =0, inference is altered if π > 1-r # /r xy. Therefore 1-r # /r xy defines the threshold for replacement: TR( r xy =0) Assumes means and variances are constant across samples, alternative calculations available.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Thresholds for Sample Replacement: Toy example Set R =r # and solve for r xy : If half the sample is replaced (  =.5), original inference is invalid if r xy < 2r # -r xy Therefore, 2r # -r xy defines the threshold for replacement: TR(  =.5 ): If threshold = r # =.4 and r xy =.6 then inference invalid if r xy = 2r # -r xy =2x.4-.6=.2.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Fundamental Problem of Inference to an Unsampled Population (External Validity) But how well does the observed data represent both populations? 9 10 11 3 4 5 888666888666 6 4

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Thresholds for Sample Replacement If r xy =0, inference is altered if π > 1-r # /r xy. Therefore 1-r # /r xy defines the threshold for replacement: TR( r xy =0) If threshold = r # =.4 and r xy =.6 then inference invalid if % replaced = π =1-r # /r xy =1-.4/.6=1/3.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Fundamental Problem of Inference and Approximating the Counterfactual with Observed Data (External Validity) 9 10 11 3 4 5 66 6 666 6 6 64 How many cases would you have to replace with cases with zero effect to change the inference? Assume threshold is: δ # =4: 1- δ # / =1-4/6=.33 =(1/3) 6 6 0

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Empirical example of Thresholds for Replacement: Effect of Open Court (Borman et al) TR(  =.5)= 2r # -r xy|z =2(.29)-.54=.03. Correlation between Open Court and reading achievement would have to be less than.03 to invalidate inference if half the classrooms in the sample were replaced (e.g., with classrooms with no effect). TR(r xy =0)= 1-r # /r xy|z =1-(.29/.54)=.47 About 47% (abut 23 classrooms) would have to be replaced with others for which Open Court had no effect (r xy =0) to invalidate the inference in a combined sample.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Review of Correlational Framework Statistical control  impact of a confound Internal validity: Impact necessary to change an inference External validity: unobserved correlations necessary to change an inference

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Exercise 4: Robustness for Sample Representativeness (External Validity) 1)Identify a statistical inference in your own work or in the literature for which there is concern about the external validity 2) Identify possible populations for which the effect may not apply 3) Note the t-ratio and sample size 4) Calculate robustness of inference using http://www.msu.edu/~kenfrank/papers/calculating%20indic es%203.xls How do the indices change when you change the sample size? t-ratio? 5) Discuss with a new partner your inference and how robust you think it is. Partner can challenge. Then change roles.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Thresholds for Sample Replacement: Toy example Set R =r # and solve for r xy : If half the sample is replaced (  =.5), original inference is invalid if r xy < 2r # -r xy Therefore, 2r # -r xy defines the threshold for replacement: TR(  =.5 ): If threshold = r # =.4 and r xy =.6 then inference invalid if r xy = 2r # -r xy =2x.4-.6=.2.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Conclusion Applications of frameworks Limitations There will be debate about inferences: quantify Assumptions as the bridge between statistics and science

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Applications of Frameworks % bias to invalidate Any estimate, threshold Counterfactual Good for experimental settings (treatments) Think in terms of replacement of cases Correlational Good for continuous predictors Think in terms of correlations Both can be applied to Internal and External Validity

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Limitations on Inference Without random sample and random assignment causal inferences are uncertain. They are inferences. Observation study Internal validity: omitted variables Randomized Study External validity: fidelity, attrition Paradox of external validity If results of randomized study influence behavior, then inherent limit on external validity Prior to study, subjects sign up because of fit, political pressure, etc After study, subjects sign up because they think it generally works Inferences will be debated based on differences in Experience Power Goals

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum It’s all in how you talk about it! Do best method you can Include relevant controls! But science is as much in the nature of the discourse as the method Virtue epistemology Greco, 2009; Kvanig, 2003; Sosa, 2007 What would it take to invalidate the inference? Frank, K.A., Duong, M.Q., Maroulis, S., Kelcey, B. under review. Quantifying Discourse about Causal Inferences from Randomized Experiments and Observational Studies in Social Science Research

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Quantify what it Would Take to Change the Inference Objections to moving from statistical to causal inference –No unobserved confounding variables –Treatment has same effect for all Robustness indices quantify how much must assumptions must be violated to invalidate inference. No new causal inferences! –robustness indices merely quantify terms of debate regarding causal inferences. Can be used with any threshold. Basically, characterizing size of p-value or t-ratio, instead of “barely significant” versus “highly significant” Can be used (theoretically) for any t-ratio –Discuss: Statistical inference as threshold? Difficult to defend tenuous effects (e.g., p =.049).

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Assumptions are the bridge between statistical and causal inference Statistical Inference Causal Inference Assumptions Cornfield, J., & Tukey, J. W. (1956, Dec.), Average Values of Mean Squares in Factorials. Annals of Mathematical Statistics, 27(No. 4), 907_949.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum In Donald Rubin’s words “Nothing is wrong with making assumptions; on the contrary, such assumptions are the strands that join the field of statistics to scientific disciplines. The quality of these assumptions and their precise explication, not their existence, is the issue”(Rubin, 2004, page 345). Ken adds: and we should talk about the inferences in terms of the sensitivity to the assumptions

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Paul Holland and Don Rubin This [the use of randomization to alleviate the problems of untestable assumptions] should not be interpreted as meaning that randomization is necessary for drawing causal inferences. In many cases, appropriate untestable, assumptions will be well supported by intuition, theory or past evidence. In such cases we should not avoid drawing causal inferences and hide behind the cover of uninteresting descriptive statements. Rather we should make causal statements that explicate the underlying assumptions and justify them as well as possible.”

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal validity example: kindergarten retention External validity exampleOpen Court curriculum Extensions of the framework Correlational Framework Impact of a Confounding variable Internal validity Example: Effect of kindergarten retention External validity example: effect of Open Court curriculum Emails of Users 'Crosnoe, Robert' ; mseltzer@ucla.edu; loyunjia@msu.edu; rwerum2@unl.edu Muller, Chandra L ; bollen@unc.edu; simon.cheng@uconn.edu; drew@drewdimmery.com; bradley.blaylock@okstate.edu; fabiog@ntu.edu.sg; shevlin@u.washington.edu; ben.b.hansen@umich.edu; chosman@umich.edu; paul.holland200@att.net; efjuncai@cityu.edu.hk; fbrykho@cityu.edu.hk; mjs3@princeton.edu; gabriela.olivares@uni.edu; klugman@temple.edu; mgaddis@live.unc.edu; Arthur.Kraft.1@city.ac.uk; HuaiZhang@ntu.edu.sg; jeeseonkim@wisc.edu; michael.weiss@mdrc.org; mgangl@ssc.wisc.edu; morgan@cornell.edu; dharding@umich.edu; larcker_david@gsb.stanford.edu; trusticus@london.edu; tad61@columbia.edu; mfeng@katz.pitt.edu; chanli@katz.pitt.edu; gpainter@usc.edu; mcorreia@london.edu; jcore@mit.edu; ksmith1@unl.edu; ssbsup@okstate.edu; Spiro Maroulis ; 'Ben Kelcey' ; 'Minh Duong' ; mseltzer@ucla.edu; cwinship@wjh.harvard.edu; 'James Moody' ; 'William Carbonaro' ; 'Mark Berends' ; Min Sun ; Richard.Taffler@wbs.ac.uk; machteld.vandecandelaere@ppw.kuleuven.be; annowens@usc.edu; keongkim@cityu.edu.hk; smithas@rhsmith.umd.edu; jwhitaker@richmond.edu; pkr@pkr.in; eminor1@nl.edu; r- korajczyk@kellogg.northwestern.edu; R.H.Cuijpers@tue.nl; liuyqphilly@gmail.com; ranxu@msu.edu; sarahgaley@gmail.com; toveave@gmail.com; chentin4@msu.edu; linqinyun1001@gmail.com; aaron.samuel.zimmerman@gmail.com; torphyka@gmail.com; 'loyunjia@msu.edu; ichiench@msu.edu; croninge@umd.edu; djklasik@umd.edu; David Williamson Shaffer ; dkaplan@education.wisc.edu; Eric Camburn ; Geoffrey Borman ; georgek@uic.edu; stephen.morgan@jhu.edu; Cavanagh, Shannon E ; Dedrick, Robert ; John Lockwood ; Lou Mariano ; Joshua Cowen ; pay2n@virginia.edu; T Gmail ; Yu Xie ; smithas@rhsmith.umd.edu; Konstantopoulos, Spyros ; Kimberly S. Maier ; William Schmidt ; Houang ; reckase@msu.edu; knaebleb@uwstout.edu; dutters@uwstout.edu; mcavoy@npg.wustl.edu; sgupta@msu.edu; ricklaux@psu.edu; dlynch@bus.wisc.edu; JMathieu@business.uconn.edu; deshon@msu.edu; carms@wharton.upenn.edu; d.petmezas@surrey.ac.uk Last sent april 6 2015, 9:15 a.m.

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal.

Similar presentations

Presentation on theme: "Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal.

Similar presentations

Presentation on theme: "Replacement Cases Framework Conclusion Correlational Framework overview Thresholds for inference and % bias to invalidate The counterfactual paradigm Internal."— Presentation transcript:

Similar presentations

About project

Feedback