1. Evaluating the performance of advanced causal inference methods in real world data through large-scale replication of randomized controlled trials Jessica Franklin Assistant Professor of Medicine Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine Brigham and Women’s Hospital, Harvard Medical School, Boston July 28, 2019
© 2018 Harvard / Brigham Division of Pharmacoepidemiology

2. Disclosures
Replication of 30 published RCTs is funded by FDA (HHSF C).
Additional FDA funding to replicate 7 ongoing RCTs.
This presentation reflects the views of the authors and should not be construed to represent FDA’s views or policies.
© 2018 Harvard / Brigham Division of Pharmacoepidemiology

3. Effectiveness Research with Healthcare Databases
RCT data
Non-interventional data
90%
10%
Research data
Transactional data
Data collected PRIMARILY for research
Data used SECONDARILY for research
For purpose
Other purpose
Other purpose
Data specifically for study purpose
Data intended for other studies
Clinical documentation
Administrative
Example
Framingham Study
Cardiovas Health Study
Slone Birth Defects Study
Some registries
Nurses’ Health Study 1
EHR-based studies
NDI linkage
Lab test databases
Claims data studies
Geocoding/census
Background
Database Studies
Franklin J, Schneeweiss S CPT 2017

4. Effectiveness Research with Healthcare Databases
RCT data
Non-interventional data
90%
10%
Research data
Transactional data
Data collected PRIMARILY for research
Data used SECONDARILY for research
For purpose
Other purpose
Other purpose
Data specifically for study purpose
Data intended for other studies
Clinical documentation
Administrative
Example
Framingham Study
Cardiovas Health Study
Slone Birth Defects Study
Some registries
Nurses’ Health Study 1
EHR-based studies
NDI linkage
Lab test databases
Claims data studies
Geocoding/census
Background
Database Studies
Franklin J, Schneeweiss S CPT 2017

5. Comparing RCT and RWE results
Improved understanding of methodology for nonrandomized RWE studies over the last 20 years.
Major concerns about the validity of nonrandomized RWE remain.
If RWE findings can match RCT findings, then we gain confidence in RWE studies.
Can help us learn:
Which methods perform better in real data and real clinical questions
Which questions can be answered with RWE
Background

6. Several prior literature reviews compared published RWD analyses and RCTs
Background

7. Several prior literature reviews compared published RWD analyses and RCTs
RCTs and RWD focus on different populations
Published RWE studies are of varying methodological quality
RCTs and RWE that are not significant may be suppressed
If RCT results are published first, RWD studies that conflict may be suppressed
Methodology for comparing RCT and RWD results is problematic
Background

8. RCT DUPLICATE Phase 1 Study Goals Phase 2 Phase 3
© 2018 Harvard / Brigham Division of Pharmacoepidemiology

9. RCT DUPLICATE Three claims data sources: Optum Clinformatics
Phase 1
Three claims data sources:
Optum Clinformatics
Truven Marketscan
Medicare
Phase 2
Study Goals
Phase 3
© 2018 Harvard / Brigham Division of Pharmacoepidemiology

10. What constitutes success?
Identify a space in terms of indications and outcomes where we have confidence that non-randomized designs using healthcare databases can support regulatory decision making
Confirm the design and analytic choices that make healthcare database studies interpretable for decision making (causal conclusions)
Develop a RWE study implementation process that promotes transparency and reproducibility of analyses
Study Goals

11. Overall search strategy
Looking for a mix of:
Active vs placebo comparators
Large vs small effects
Superiority vs non-inferiority
Different clinical areas
40 RCTs
Positive findings
Negative findings
Database of secondary approvals
Database of primary approvals
ClinicalTrials.gov
33 replicable studies
7 replicable studies
Suggestions from FDA
Initial search
Final selection
RCT Search Strategy

12. Number of RCTs Number of RCTs Number of RCTs
Indication
Number of RCTs
Pre-marketing 17
Post-marketing 23
Therapeutic Area
Number of RCTs
Anti-diabetic medications 7
Antiplatelets
Novel oral anticoagulants 6
Antihypertensives 4
Anti-osteoporosis medications
Asthma medications
Chronic obstructive pulmonary disease medications 3
Heart failure
Statins 1
Antiarrhythmic
RCT Search Strategy
Comparator
Number of RCTs
Active comparison 18
Placebo + standard of care comparison 22

13. Trial type and findings
Trials powered for superiority
Number of RCTs 
Successful superiority (1a) 19
Failed superiority with non-inferiority (2a) 2
Failed superiority – No non-inferiority margin specified (3) 4
Trials powered for non-Inferiority
Unintended superiority (4a) 7
Successful non-inferiority (5a)
Failed non-inferiority (6) 1
RCT Search Strategy
*NI = Non-inferiority; S = Superiority

14. Implementation goals Make the process transparent and informed
Ensure that the final decision to undertake a study is informed only by study power and patient characteristics, not by study results.
Ensure that specific designs and analyses are selected based on science, not based on study results.
RWD Implementation Process

15. Process overview
RWD Implementation Process

16. Process overview
RWD Implementation Process
Trial selection

17. Process overview RWD Implementation Process Trial selection
Similar protocol used for all replications
Designate one analysis as primary
Implement in multiple databases where possible
Seek IRB approval

18. Process overview RWD Implementation Process
Check power/covariate balance.
Refine design
Trial selection
Similar protocol used for all replications
Designate one analysis as primary
Implement in multiple databases where possible
Seek IRB approval

19. © 2018 Harvard / Brigham Division of Pharmacoepidemiology

20. © 2018 Harvard / Brigham Division of Pharmacoepidemiology

21. Process overview RWD Implementation Process
Check power/covariate balance.
Refine design
Trial selection
Outcome analyses begin
Similar protocol used for all replications
Designate one analysis as primary
Implement in multiple databases where possible
Seek IRB approval

22. Process overview RWD Implementation Process Document all findings
Check power/covariate balance
Refine design
Trial selection
Outcome analyses begin
Similar protocol used for all replications
Designate one analysis as primary
Implement in multiple databases where possible
Seek IRB approval

23. Limitations
When replicating published RCTs, findings are known in advance
Ability to replicate depends on choices of investigators
Even when replicating inclusion/exclusion criteria, populations, drug adherence, follow-up time will be different.
We will attempt to understand post-hoc whether these factors caused lack of agreement between findings.
Focused on health insurance claims
Use of EHR or other RWD sources would lead to selection of a different set of RCTs and possibly different conclusions.
© 2018 Harvard / Brigham Division of Pharmacoepidemiology

24. Thanks! Collaborators at FDA
David Martin, Robert Temple, Mark Levenson, and others
External Advisory Board
Alan Brookhart, Steve Goodman, Miguel Hernan, Wayne Ray, Samy Suissa
Collaborators at HMS
Sebastian Schneeweiss, Robert Glynn, Elisabetta Patorno, Krista Huybrechts, Josh Gagne, Seoyoung Kim, Brian Bateman, and others
© 2018 Harvard / Brigham Division of Pharmacoepidemiology