Statistical Methods for Observational Studies Lecture 4 (of 4) Steve Fienberg Memorial Lectures Series in Advanced Analytics November 2018 Statistical Methods for Observational Studies Lecture 4 (of 4) Dylan Small University of Pennsylvania Slides posted at my web site: www-stat.wharton.upenn.edu/~dsmall

Bracketing vs. Synthetic Control

Bracketing Model

Example where synthetic control is biased

Bracketing vs. synthetic control continued

Reducing ambiguity in an observational study An association between the treatment and outcome after controlling for measured confounders is ambiguous, it could reflect a real treatment effect or a hidden bias.  Typical observational study: Addresses ambiguity by just mentioning possibility that hidden bias invalidates the study’s conclusion in the discussion section. An easy way to publish false causal conclusions is to decline to look for evidence that might reveal bias if present. Quasi-experimental devices: Enlarge the set of considered associations in an effort to reduce ambiguity. Devices are often selected to reduce ambiguity from particular plausible rival hypotheses. In spirit of Popper’s philosophy that a theory in the empirical sciences can never be proven but it can be falsified, meaning that it can and should be scrutinized by tests that might falsify it.

Replication and reducing ambiguity Replication of an observational study in which treatments are chosen in a similar way does not reduce ambiguity. Replication of an observational study in which treatments are chosen in different way can reduce ambiguity.

Example of replication

Evidence factors

Go to “Using Split Samples and Evidence Factors in an Observational Study of Neonatal Outcomes”: Zhang, Small, Lorch, Srinivas and Rosenbaum (2011)

Regional vs. General Anesthesia

Elaborate Theories

Elaborate Theory for Smoking and Lung Cancer

Weighing evidence Cochran (1965): “The combined evidence on a question that has to be decided mainly from observational studies will usually consist of a heterogeneous collection of results of varying quality, each bearing on some consequence of the causal hypothesis … [The investigator] cannot avoid an attempt to weigh the evidence for and against, since some results are so vulnerable to bias that they should be given low weight even if supported by routine tests of significance.”

Evidence for smoking causing lung cancer

Evidence for hormone replacement therapy

Summary of Lectures Observational studies can be made more reliable by Designing the study and specifying a protocol before looking at estimates of treatment effects. Designs that probe for sources of hidden bias and test rival explanations for why there is an association between treatment and outcome other than the treatment having a causal effect. Proper measures of weight of evidence through sensitivity analysis and control for multiple testing. Cochran (1972), “In conclusion, observational studies are an interesting and challenging field which demands a good deal of humility, since we can claim only to be groping toward the truth.”

Observational Studies and Steve Fienberg’s Legacy Observational studies are a way of trying to improve society through using data and statistical and scientific reasoning to figure out what policies and treatments work. Using statistics as a force for good and improving society was Steve Fienberg’s life’s work.

Steve (Statistical Science interview): “I guess I’d like to be remembered as somebody who produced really good students and who helped change the image of statistics in the sense that lots of people now work on serious applied problems and help solve them.” Steve stood for using reason, in particularly statistical reason, to improve society. He would have been so saddened at the lack of reason, the bigotry, that contributed to the murder of his wife and 10 others at Tree of Life Synagogue. But we should keep building on Steve’s life work, to bring statistical reasoning to bear on the problems of society, to keep improving society and to reduce reasonless deaths.