1. Alternatives to Randomized Trials for Determining Treatment Efficacy (or Harm) Thomas B. Newman, MD, MPH

2. Outline n Background n Instrumental variables/natural experiments n Measuring unrelated variables to estimate bias n Statistical adjustment --Febrile Infant Study n Propensity scores n Other designs

3. Background n Why do RCTs? –Assemble comparable groups (avoid confounding) –Allow blinding (to avoid placebo effect, cointerventions, and bias in measuring outcome variable) n Observational studies –May be able to assemble comparable groups or use statistical adjustment –Won’t be blinded

4. Why is it hard to assemble comparable groups without randomizing? n By comparable, we mean comparable with respect to prognosis/risk of outcome being studied. n Treated people will often be at higher risk of bad outcome (confounding by indication for treatment). n For some screening tests, screened people may be at lower risk (volunteer bias)

5. When it’s easy n For outcomes not related to indications for treatment there can’t be confounding by indication for treatment n Example: even without RCT, easy to attribute rhabdomyolysis to statins. But harder to tell effects on stroke without RCT.

6. Natural Experiments and Instrumental Variables n Find a time or place where receipt of treatment was more-or-less random (at least unlikely to be related to prognosis) –E.g., time-series analyses where something changed (e.g. new intervention became available) n Instrumental variables: measurable factors that influence treatment not otherwise associated with outcome

7. Use of large databases n Basic idea: overcome lots of random error with large sample size n Allows use of (weak) surrogate measures for actual predictor n Predictable effect on results (bias toward the null) can be estimated algebraically

8. Example 1: Delayed Effects of the Military Draft on Mortality*: 1 n Origin of study: Agent Orange concern n Design: “Randomized natural experiment”using the draft lottery n Data source: computerized death certificate registries, CA and PA n Predictor variable of interest: military service *Hearst N, Newman TB, Hulley SB. NEJM 1986; 314:620-24

9. Delayed Effects of the Military Draft on Mortality: 2 n Predictor variable measured: draft lottery number below cutoff (based on date of birth)

10. BUT: Having an eligible number was a poor measure of military service:

11. Algebraic Correction: n Assume death rates are weighted averages of rates among those serving and not serving n Algebraically get risk ratio for service from risk ratio for eligibility, given the % that served in each group n Draft eligibility is an instrumental variable: associated with predictor of interest, but not with outcome

12. Results

13. Example 2: Does intensive treatment of AMI in the elderly reduce mortality?* n Study design: Linked claims and administrative data from HCFA n Distance from “intense” hospital as an instrumental variable –Distance from “intensive”hospital associated with different rates of cardiac catheterization –Distance NOT associated with different rates of comorbidity *McClellan M et al. JAMA 1994;272:859-66

14. Results: n Otherwise similar groups with higher probability of catheterization (due to lower distance from an intensive hospital) had lower mortality n Absolute effect size: about 10% reduction in mortality for catheterization within the first 7 days.

15. Summary/other examples n If variables known NOT to be associated with outcome are associated with treatment of interest, consider this approach. n Generalizes to many”natural experiments.” –E.g., an intervention is intermittently available, or only available to certain groups. -- different outcome by day of the week, etc.

16. Unrelated variables to estimate bias or confounding n One approach to selection bias: estimate rather than eliminate it n Measure an outcome that WOULD be affected by bias, but not by intervention or n Measure a predictor that would cause the same bias as the predictor of interest (and see if it does)

17. Case control study of screening sigmoidoscopy n Possible bias: patients who agree to sigmoidoscopy are likely to be different n Solution: compare benefit for cancers within and beyond the reach of the sigmoidoscope Selby et al, NEJM 1992;326:653-7

18. Effect of British “breathalyser” crackdown n Abrupt drop in accidents occurring during weekend nights (when pubs open) n No difference in accidents occurring during other hours See Cook and Campbell: Quasi- Experimentation.Boston:Houghton Mifflin, p. 219

19. Fenoterol and death from asthma n Case-control studies suggest increase risk of death in users of fenoterol n Obvious risk of confounding by intention to treat -- worse asthma, more beta- agonist use n But no association with use of oral or inhaled steroids and no or smaller association with albuterol Spitzer et al., NEJM 1992;326:501-6; Crane et al., Lancet 1989;1:917-22

20. Calcium Channel Blockers (CCB) and AMI n Population based case-control study at Group Health n Progressive increase in risk with higher doses of CCB (P <0.01) n Progressive decrease in risk associated with higher doses of beta-blockers (P =0.04) Psaty et al., JAMA 1995;274:620-25

21. Statistical adjustment: Urine Testing and UTI in the PROS Febrile Infant Study n RQ: what tests should be done to evaluate infants < 3 months old with fever? n Study design: cohort study in the AAP’s Office-based research network n Subjects enrolled by 579 different pediatricians n About half (55%) of the infants had urine tested

22. Subjects n T > 38.0 in office or in previous 24 hr at home. n Age < 3 months n Initially seen by a PROS Practitioner n Enrollment March, 1995 to March 1998 n N = 3,066

23. Urine Testing by Age and Office Temp

24. Multivariate Predictors of UTI

26. Propensity Scores -1 n Big picture: want to know if association between treatment and outcome is CAUSAL n Recall competing explanation = confounding by indication for treatment: –Factor must be associated with outcome –Factor must be associated with treatment n Traditional approach: adjust for factors associated with outcome

27. Propensity Scores -2 n Create a new variable, propensity to be treated with the intervention n Then either match on that variable or include it in multivariate analyses n Advantage: many studies have relatively few outcomes, so less power to identify potential confounders. Receipt of the intervention is much more common, so better power to identify predictors of it.

28. Example: Aspirin use and all-cause mortality among patients being evaluated for know n or suspected Coronary Artery Disease* n RQ: Does aspirin reduce all-cause mortality in patients with coronary disease n Design: Cohort study n Subjects: 6174 consecutive patients getting stress echocardiograms n Predictor: ASA use n Outcome: All-cause mortality JAMA 2001; 286: 187

29. Analysis using Propensity Scores n Two multivariable analyses: –Predictors of death –Predictors of aspirin use n Predictors of ASA use turned into a propensity score estimating probability of ASA use n Patients matched on ASA propensity score n ASA propensity score also used in multivariate model

30. Survival in Propensity-Matched Patients

31. Summary n Main threat to observational studies of treatment is confounding n Confounders are assoc. with both predictor and outcome n Instrumental variables are associated with predictor, but not (independently) with outcome n Propensity scores allow adjustment for association with predictor n If you can’t avoid bias, measure ot