1. Randomized vs. Observational Studies: Strengths and Weaknesses
William S. Weintraub, MD

2. COI: None

3. Ever Increasing Healthcare Data

4. Characteristics of Big Data : Volume
By 2015, average hospital will generate 2/3 petabyte (665 terabytes) of patient data per year.

5. Characteristics of Big Data: Veracity
Can Big Data be Meaningful in Health Care
How Can We Make Sense of All This Data
Can We Use Big Data to Make
Better Health Care Decisions

6. Veracity Can We Use Big Data for Comparative Effectiveness?
Big data is not necessary good data
Accuracy is necessary but not sufficient
Bias: Misclassification and Selection
Size does not over overcome bias 6

7. Data From EHRs Swiss Cheese Data
Missing Data Cannot be Assumed to be Missing at Random
Missing Data Does Create Bias
Swiss Cheese Data

8. Can We Use Big Data for Comparative Effectiveness to Improve Quality?
We have been comparing therapeutic and diagnostic strategies for decades
Most randomized trials have been for registration purposes
Questions concerning therapeutic and diagnostic strategies that are of societal interest may not have been addressed
Thus, there are gaps in our knowledge concerning medical decisions
Can observational studies from big data fill the gap?

9. The Advantage of Randomized Trials
The Randomized Controlled Trial Is The Only Method Which Can Overcome Treatment Selection Bias When Comparing Therapeutic or Diagnostic Strategies

10. Then Why Do We Need Non-Randomized Approaches
Disadvantages of Randomized Trials:
Lack of generalizability
Expense
Become outdated
Crossovers
Non-blinding of many studies
Defining the groups to be compared
Limited power in subgroups

11. The Problem of Generalizability
Typical trial cost ~$4.5 m
Typical registry costs ~$1.5 M
Decision-makers are clinicians, payers, regulators 19

13. Advantages and Disadvantages of Observational Data for CEA
Large sample sizes give great power
Ability to examine subgroups
Real world patients
More contemporary data
Disadvantages:
Treatment selection bias
Data quality
Non-adjudicated outcomes
Uncertain definition of treatment groups
Covariate and outcome data of interest not available

14. Methods to Reduce Treatment Selection Bias
Rigorously defined groups
Multivariate analysis
Propensity Score Analysis
Subgroups defined by propensity score
Propensity score used as a covariate in multivariate analysis
Matched groups by propensity score
Inverse probability weighting
Instrumental Variable
But, you cannot correct for what you do not measure!
And, size does not overcome bias!

15. Risk of local adverse events following cardiac catheterization by hemostasis device use - phase II.
Tavris DR, Dey S, Albrecht-Gallauresi B, Brindis RG, Shaw R, Weintraub W, Mitchel K.
Embedded study in the National Cardiovascular Data Registry.
Data from 59 institutions and 13,878 procedures performed during the last quarter of 2003.
Additional prospective data collection
Multivariable analysis was used to assess the risk associated with type of device and gender, controlling for demographic and physiologic variables, type of procedure, and comorbidity.
Serious adverse events in 3.37% of patients, the most common being bleeding with hematoma (2.00%). VasoSeal, demonstrated a high risk of any vascular complication compared to manual compression controls (OR = 2.38, p = ).
J Invasive Cardiol. 2005;12:

16. NCDR CathPCI: Patient Characteristics
DES (217,675) vs BMS(45,025)
Unadjusted
IPW Adjusted
DES
BMS
DES
BMS
Age
74.5
75.3*
74.7
74.8
Female
43%
40%*
43%
43%
Caucasian
90%
91%*
90%
90%
Diabetes
 32%
32%
 32%
32%
Renal Failure
 6%
8%*
 7% 7%
Hypertension
80%
80%*
80%
81%
Prior PCI
28%
26%*
28%
28%
Prior CABG
22%
28%*
23%
23%
Urgent Status
38%
36%*
37%
38%
STEMI
10%
16%*
11%
11%
Douglas P et al LBCT ACC 2009

17. NCDR: DES and BMS: 30-mo Event Rates Adjusted25
BMS 20
DES 15
Rate / 100 patients 10 5
Death MI
Revasc
Bleeding
Stroke
HR = 0.75
(0.73,0.77)
HR = 0.76
(0.72,0.80)
HR = 0.91
(0.89,0.94)
HR = 0.91
(0.85,0.98)
HR = 0.96
(0.88,1.04)
Douglas P et al LBCT ACC 2009

19. Supported by a grant from the National Heart, Lung, and Blood Institute (#RC2HL101489) and by an Institutional Development Award from the National Institute of General Medical Sciences (#U54-GM104941) of the National Institutes of Health.

20. Data from 644 Sites
NCDR Sites
STS Sites

22. Baseline Data

23. 30-Day
1-Year
2-Year
3-Year
4-Year
30-Day
1-Year
2-Year
3-Year
4-Year
16.4%
( )
12.4%
( )
8.98%
( )
6.24%
( )
2.25%
( )
16.0%
( )
12.1%
( )
8.76% (
6.00%
( )
2.07%
( )
CABG mortality
20.8%
( )
15.9%
( )
11.3%
( )
6.55%
( )
1.31%
( )
20.9%
( )
16.0% )
11.2%
( )
6.36%
( )
1.21%
( )
PCI mortality
Relative
Risk
1.72
( )
0.94
( )
0.78
( )
0.76
( )
0.76
( )
1.72
( )
0.95
( )
0.79
( )
0.78
( )
0.79
( )

25. Coronary Artery Bypass Grafting vs Percutaneous Coronary Intervention and Long-term Mortality and Morbidity in Multivessel Disease Meta-analysis of Randomized Clinical Trials of the Arterial Grafting and Stenting Era
Ilke Sipahi, MD; M. Hakan Akay, MD; Sinan Dagdelen, MD; Arie Blitz, MD; Cem Alhan, MD
JAMA Intern Med. 2014;174:

26. Observations Data for Comparative Effectiveness
Observational studies can provide real-world outcomes with greater generalizability than randomized trials
Linking robust clinical databases with administrative database capitalizes on the advantages of both
This allows for very large studies with power to examine subgroups
Administrative databases can also supplement clinical databases with resource use/cost data
Both clinical and cost comparative effectiveness studies can be done
The major limitation of observational studies is treatment selection bias
For comparative effectiveness to reach is potential, randomized trials and observational studies will both have critical roles to play