1. Is a “Discussion” on “Are Observational Studies Any Good” Any Good Don Hoover May 2, 2014 1

2. Everyone Already Knows Observational Studies Are Not Perfect … Right? But who thinks – the real type 1 error is 0.55 when the nominal is 0.05? – The real coverage of a 95% confidence interval is 25%? – That’s what David Madigan and the OMAP team find This obviously makes such results meaningless But how many papers with these properties are being (and will continue to be) published ??? 2

3. But Does David’s Talk Really Apply to ALL Observational Studies? They Only Look at Observational Studies of Drug Use and Adverse Consequences There’s other kinds of Observational Studies … on HIV, Epi, Health Behaviors, Nutrition, etc. – No one has looked at these types of studies These other studies must have similar problems Maybe at a smaller magnitude – But there are no “negative controls” for these settings … so no one can check this 3

4. The Approach here is Creative and Innovative Finding Negative Control Exposures or Outcomes to derive empirical distribution of the test statistic somewhat equalizes assumptions and unmeasured confounding With a given Drug Use as the exposure and a given Disease the outcome, such negative controls are readily available in many data sets So maybe something like it should be used when possible But now some questions …… 4

5. Q1- Why were Negative Control Drugs More Associated With Outcomes than by Chance? People put on Any Drug are Sicker? Those receiving a negative (control) drug are more likely to receive some other positive drug? Those apriori more likely to have a given disease outcome are steered to the negative drugs? Incorrect statistical models used? 5

6. Q2- Is this Approach Practical? A lot more work to fit many models than the standard approach which only fits one – More money as well - A grant application using it would be less likely to get funded – More work also means more chance for error in implementation 6

7. Q3 – How does one interpret a positive drug with empirical P < 0.05? 7 Positive Drug with empirical P < 0.05 The use of an “empirical” approach acknowledges we do not know what is going on so maybe the P < 0.05 is from model artifact not causal Calibrated Normal Scores of Negative Controls

8. Q4 – What is done with “Negative Drugs” more extreme than the Positive One 8 Calibrated Normal Scores of Negative Controls Positive Drug with P < 0.05 Should these Negative Controls all be Examined for Causal Association as their Signal is larger than the positive drug?

9. Q5 - How to handle Heterogeneity in Denominator of Calibration Statistic 9 Variance may introduce Apples to Oranges comparisons especially if although such does not appear to be the case in the examples David used From Schumie … Madigan Stat Med 2014 33; 209-18