1. Statistical Methods for Observational Studies Lecture 4 (of 4)
Steve Fienberg Memorial Lectures Series in Advanced Analytics
November 2018
Statistical Methods for Observational Studies Lecture 4 (of 4)
Dylan Small
University of Pennsylvania
Slides posted at my web site: www-stat.wharton.upenn.edu/~dsmall

2. Bracketing vs. Synthetic Control

3. Bracketing Model

4. Example where synthetic control is biased

5. Bracketing vs. synthetic control continued

6. Reducing ambiguity in an observational study
An association between the treatment and outcome after controlling for measured confounders is ambiguous, it could reflect a real treatment effect or a hidden bias. 
Typical observational study: Addresses ambiguity by just mentioning possibility that hidden bias invalidates the study’s conclusion in the discussion section.
An easy way to publish false causal conclusions is to decline to look for evidence that might reveal bias if present.
Quasi-experimental devices: Enlarge the set of considered associations in an effort to reduce ambiguity. Devices are often selected to reduce ambiguity from particular plausible rival hypotheses.
In spirit of Popper’s philosophy that a theory in the empirical sciences can never be proven but it can be falsified, meaning that it can and should be scrutinized by tests that might falsify it.

7. Replication and reducing ambiguity
Replication of an observational study in which treatments are chosen in a similar way does not reduce ambiguity.
Replication of an observational study in which treatments are chosen in different way can reduce ambiguity.

8. Example of replication

9. Evidence factors

12. Go to “Using Split Samples and Evidence Factors in an Observational Study of Neonatal Outcomes”: Zhang, Small, Lorch, Srinivas and Rosenbaum (2011)

13. Regional vs. General Anesthesia

15. Elaborate Theories

16. Elaborate Theory for Smoking and Lung Cancer

17. Weighing evidence
Cochran (1965): “The combined evidence on a question that has to be decided mainly from observational studies will usually consist of a heterogeneous collection of results of varying quality, each bearing on some consequence of the causal hypothesis … [The investigator] cannot avoid an attempt to weigh the evidence for and against, since some results are so vulnerable to bias that they should be given low weight even if supported by routine tests of significance.”

18. Evidence for smoking causing lung cancer

19. Evidence for hormone replacement therapy

21. Summary of Lectures Observational studies can be made more reliable by
Designing the study and specifying a protocol before looking at estimates of treatment effects.
Designs that probe for sources of hidden bias and test rival explanations for why there is an association between treatment and outcome other than the treatment having a causal effect.
Proper measures of weight of evidence through sensitivity analysis and control for multiple testing.
Cochran (1972), “In conclusion, observational studies are an interesting and challenging field which demands a good deal of humility, since we can claim only to be groping toward the truth.”

22. Observational Studies and Steve Fienberg’s Legacy
Observational studies are a way of trying to improve society through using data and statistical and scientific reasoning to figure out what policies and treatments work.
Using statistics as a force for good and improving society was Steve Fienberg’s life’s work.

23. Steve (Statistical Science interview): “I guess I’d like to be remembered as somebody who produced really good students and who helped change the image of statistics in the sense that lots of people now work on serious applied problems and help solve them.”
Steve stood for using reason, in particularly statistical reason, to improve society.
He would have been so saddened at
the lack of reason, the bigotry, that
contributed to the murder of his wife
and 10 others at Tree of Life Synagogue.
But we should keep building on Steve’s
life work, to bring statistical reasoning to
bear on the problems of society, to keep
improving society and to reduce reasonless
deaths.