1. Pragmatic Clinical Trials: Challenges with Clinical Event Ascertainment using RWD
Frank W. Rockhold, PhD
Professor of Biostatistics and Bioinformatics
Duke University Medical Center
Duke/Stanford CEC Summit
September 26-27, 2018, Chicago, Il

2. Disclosure Statement – Frank W Rockhold, PhD
Research Funding: NIH, PCORI, Duke Clinical Research Institute, Astra Zeneca, ReNeuron, Luitpold, Alzheimer's Drug Discovery Foundation, Janssen, BMS
Consulting and Honoraria:
Boards: Frontier Science Board of Directors, DataVant Scientific Advisory Board, European Medicines Agency Technical Advisory Group
Equity Interest: GlaxoSmithKline, DataVant

3. Topics to Discuss
Pragmatic Clinical Trials (PCTs) answer questions that are about real world effectiveness.
It can be a supplement to and not necessarily a replacement for “classic” (explanatory/confirmatory) RCT’s.
Depends on the question and inference desired
For this discussion RWD is EMR/Claims data, i.e. data that are a result of a patient health encounter
“Investigator” vs GP: Has an impact on recruitment and event ascertainment.
Is it really less expensive? In the near term the focus should be on ”how” and not “how much”.
Data management vs healthcare informatics- resource and cost shifting.
Lessons learned should be of value to all clinical trial conduct.
The approach can be useful for safety studies but there needs to be agreement on potential tradeoffs in event ascertainment.

4. Some questions to consider
What loss in ascertainment sensitivity are we willing to live with in a study with RWD study?
For safety studies of uncommon or serious events are PCT’s using RWD sources an adequate alternative to RCT’s?
OBS study is not likely to fully discharge a safety risk.
Do not abandon the concept of randomization too quickly
Given practicality considerations what should be our threshold for saying an RWD study is the “only way” even in situations where it is possible to conduct an RCT?? And what is the tradeoff in scientific credibility vs practicality?

5. PRECIS-2 (Loudon, K. , Treweek, S. , Sullivan, F. , Donnan, P
PRECIS-2 (Loudon, K., Treweek, S., Sullivan, F., Donnan, P., Thorpe, K.E., and Zwarenstein, M., The PRECIS-2 tool: designing trials that are fit for purpose. BMJ, : p. h2147 )

6. Designing a randomised pragmatic clinical trial
Ways that Randomised Controlled Trials (RCTs) and PCTs can differ
“Classic” RCT
P”R”CT
Intentionally homogeneous to maximise
treatment effect
Randomisation and blinding
Clinical measures, intermediate endpoints,
composite endpoints, clinical outcomes
Protocol defines the level and timing of testing.
Physicians blinded to data
Fixed standard of care or placebo
Conducted only by investigators with
proven track record
Visit schedule and treatment pathway defined
in the protocol
Patients wishing to change treatment must
withdraw from the study
Compliance is monitored closely
Close monitoring of adherence
Intent to treat, per-protocol and completers
Eligibility Criteria
Randomisation and blinding
Endpoints
Tests and diagnostics
Comparison intervention
Practitioner expertise
Follow up
Continuity
Participant compliance
Adherence to study protocol
Analysis
Heterogeneous - representative of normal treatment population
Randomisation and rarely blinding
Clinical outcomes, PROs, QoL, resource use
Measured according to standard practice
Standard clinical practice
Employment of a variety of practitioners with differing expertise and experience
Visits at the discretion of physician and patient.
Standard clinical practice – switching therapy according to patient needs
Passive monitoring of patient compliance
Passive monitoring of practitioner adherence
All patients included

7. ADAPTABLE*, the Aspirin Study – A Patient-Centered Trial
*theaspirinstudy.org

8. PCORnet seeks to improve the nation’s capacity to conduct clinical research by creating a large, highly representative, national patient-centered network that supports more efficient clinical trials and observational studies.

9. Study Design Patients with known SCVD
(ie MI, OR cath ≥75% stenosis of ≥1 epicardial vessel or PCI/CA BG)
AND ≥ 1 Enrichment Factor
Identified through EMR (computable phenotype) by CDRNs (PPRN pts. already part of a CDRN are eligible)
Pts. contacted with trial information and link to eConsent; Treatment assignment provided directly to patient
*Enrichment factors
Age > 65 years
Creatinine > 1.5
Diabetes (Type 1 or 2)
3-vessel coronary artery disease
Cerebrovascular disease and/or peripheral artery disease
EF <50% by echo, cath, nuclear study
Current smoker
Exclusion Criteria
Age < 18 yrs
ASA allergy or contraindication (including pregnancy or nursing)
Significant GI bleed within past 12 months
Significant bleeding disorder
Requires warfarin or NOAC or Ticagrelor
ASA 81 mg QD
ASA 325 mg QD
Electronic F/U Q 3-6 months;
Supplemented with EMR/CDM/claims data
Duration: Enrollment over 24 months;
Maximum f/u of 30 months
Primary Endpoint: Composite of all-cause mortality, nonfatal MI, nonfatal stroke Primary Safety Endpoint: Major bleeding complications

10. ADAPTABLE – How Pragmatic is it?

11. Centralized Enrollment & Follow-Up
Patient Portal
Info/eConsent/Randomization
Patient Reported Outcomes
Medication use
Health outcomes
Randomized to Q3 or Q6
DCRI Call Center
Patients who miss 2 contacts
Patient Reported Outcomes
Medication use
Health outcomes
ADAPTABLE
Patient 3 6 9 12 15 30
PCORNet Coordinating Center FOLLOW-UP
Via Common Data Model
Longitudinal health outcomes
Baseline Data
CMS and private health plans FOLLOW-UP
Longitudinal health outcomes

12. Information Flow
Patient
PCORnet
Supplemental Linkages
Mytrus Patient Portal
EMR
Medicare Claims
National Death Index
Private Health Plan Data
ADAPTABLE Study Database
Each data source arrives at the coordinating center via a different mechanism
All will contribute to eventual study database
Algorithm based decisions for discrepant data/event ascertainment

13. Information Asymmetry
Different participants with different sources of data contributing to endpoint ascertainment
Vary by site, Medicare & health plan coverage are not uniform
Vary by site or patient if fields are inaccurate or missing
PtP
EMR
CMS
NDI HP
Patient #1
PtP
EMR
CMS
NDI HP
Patient #2
PtP
EMR
CMS
NDI HP
Patient #3
PtP
EMR
CMS
NDI HP
Patient #4

14. Information Latency Wg
Data Source
Availability
Min – Max Delay
Participant Self-reported data
Instant
None
Electronic Records
EMR from PCORnet DataMart
Quarterly
3 – 6 months
Medicare Claims Data
Annual
1 – 12 months
National Death Index
Private Health Plan Data
End of study? Wg
Implication: complete data for the study will become available 1 year after the last patient last follow up

15. Handling Disagreement Across Different Data Sources
Patient reported hospitalizations that are not observed in EMR data will be queried via:
Medicare
fee-for-service claims
Large national health plans
(FDA’s Mini-Sentinel initiative)
DCRI
Call Center

16. Missing Follow-up DCRI Call Center Contact with the site
Patient Finder
Social Security Death Index

17. Endpoint Ascertainment and Validation
Are we capturing information completely and accurately?
A multi-faceted approach to capture outcomes
Endpoint validation and Endpoint reconciliation of MACE and Major Bleeding events
Can endpoints identified in EMR comparable to clinical endpoints confirmed by traditional adjudication process?
Events of interest: MI, stroke, major bleeding
Assess agreement: True Positive, False Positive, PPV
Provide insights on accuracy of coding algorithms and the data curation process
Could a machine algorithm help speed confirmation of events?
Absence of events from EMR is not usually verified due to low event rates
Could machine algorithms be used to look for potential false negatives?

18. Approach to Endpoint Ascertainment
Routine queries of the PCORnet CDM to capture and classify endpoints using validated coding algorithms
Hospitalizations will be identified via standardized, validated coding algorithms developed centrally and applied to the CDM
ADAPTABLE patient portal will ask about possible endpoint events (hospitalizations for myocardial infarction, stroke, or major bleeding) during participant contacts (every 3-6 months)
Such information will provide additional confirmation for the CDM- generated hospitalization data that meet criteria for the endpoints specified
Death ascertainment via Social Security Administration (Medicare beneficiaries)
LHC: Red text is not what we describe in the protocol (copied in below). We need to be consistent.
Endpoints will be ascertained by applying algorithms designed to ensure comprehensive surveillance for all potential endpoints.
•Routine queries will be applied to the PCORnet CDM to capture and classify endpoints using validated coding algorithms. Because the EHR and (by extension) the PCORnet CDM is the source of truth for the trial, no patient confirmation or other additional confirmation will be required.
•Hospitalizations for stroke, MI, or bleeding reported by patients will be evaluated in the following manner:
oQueries of the PCORnet CDM will be used to cross-check, confirm, and classify in-network hospitalizations (nonfatal MI, nonfatal stroke, or major bleeding) reported by patients.
oOut-of-network hospitalizations reported by patients and not captured in the CDM will be identified by patient self-report through the periodic electronic contacts planned throughout the study.

19. Validation Plan – Background (courtesy of Dr. Schuyler Jones)
Formal clinical endpoint classification (CEC) cannot be performed within the budget and will need to be modified in order to ensure efficiency and lower costs
Concerns exist that event ascertainment via coding algorithms of administrative claims data needs to demonstrate acceptable levels of validity and be accepted by physicians and patients alike
Conflicting data exist regarding the agreement between administrative claims data and clinical study adjudication, thus necessitating the need for this supplemental, validation plan

20. Validation Plan – Expected Number of Endpoints to Review
Random sample of approximately 20% of the MI, stroke, and major bleeding endpoints
sampled proportionally from enrolling health systems within participating CDRNs
select the first 10 MI events, 10 stroke events, and 5 bleeding events that occur within each CDRN (total=25 events per CDRN).
We will randomly select approximately 175 non-fatal MI and stroke endpoints and 50 major bleeding endpoints
Approximately 225 overall endpoints to review during the course of the study.

21. Validation Plan – Adjudication of Events
Each case randomly selected will be reviewed in a blinded manner by a disease-specific, expert adjudicator
a single cardiologist will review MI and major bleeding endpoints
a single neurologist will review stroke endpoints
Standard endpoint definitions for MI, stroke, and major bleeding will be used to adjudicate these endpoints using standardized adjudication forms

22. Validation Plan – Analysis
Formal statistical analyses will be performed to measure:
percentages of true positive results and false positive results
agreement/concordance rates for each endpoint that will be validated (non-fatal stroke, non-fatal MI, and major bleeding)
These results (if different) will not change the primary analysis of ADAPTABLE which will count those events identified via coding algorithms as described in the protocol

23. Data and Safety Monitoring
Role of a DMC in a pragmatic trial
Standard role
Additional focus: feasibility, protocol adherence, data validity
Involvement of patient representatives
If and how could the DMC make critical decisions with fragmented information during the study?
Interim analyses: differentiate a signal from noise with limited access to EMR outcome data
Differential data lag times
Benefit to risk analysis
Could a machine algorithm help even with partial data?
Is the DMC protected?
Indemnification
Source
documentation adjudication
Concordance
analyses

24. Summary: ADAPTABLE
ADAPTABLE is the largest pragmatic trial in the world to evaluate aspirin dose with expected 15,000 participants
Attempts to mimic the real world patient experience of a patient with heart disease
Collects data through different sources and employs a multi-faceted approach to capture outcomes will teach us a great deal about the value and issues with this approach in the future.
Maintained scientific rigor
randomised
active control
robust primary endpoint
ADAPTABLE will tell us a great deal about the utility of the approach to perform “mega trials” in a very different way.

25. Machine Learning for Clinical Events Classification and Safety Surveillance: An Innovation Project from the DCRI ICE-SS group
Figure 1.DCRI CEC Process
Figure 2. Euclid Trial Summary Statistics
11,082 events
Average trigger summary: 46 words (min: 1, max: 1141)
Average dossier: 1317 words (min: 1, max: 24984)
80/20/20% partition for training, validation and test, respectively. MI
BLEEDING
ALI
STROKE/TIA
DEATH
2088
3966
3019
692
1317
Yes No
Stroke Yes
TIA Yes NA
853
1235
3044
962
293
2726
317
107
214
Figure 3. Classification Results on Test Set (using trigger summary)
The mission of the Duke Clinical Events Classification Group
To provide high quality adjudicated endpoint data with scientific rigor, efficiency and innovation by coordinating and conducting Systematic, Comprehensive, Unbiased, Blinded, and Independent clinical events adjudication.
Lopes RD, et al. Not published.

26. Endpoints in Studies (Trial) with RWD
PCTs answer questions that are more real world effectiveness. Should be viewed as a supplement to and not a replacement for “classic” RCT’s.
“Investigator” vs GP: Has an impact on recruitment and event ascertainment.
Is it really less expensive? Time will tell and in short term the focus should be on ”how” and not “how much”. Cost per information unit is unknown.
Data management vs healthcare informatics- resource and cost shifting. Lessons learned should be of value to all clinical trial conduct.
Machine algorithms could likely help with EMR endpoint PPV.
The approach can be useful for safety studies but there needs to be agreement on tradeoffs in event ascertainment and effort involved in data acquisition
Even with tradeoffs of potential data gaps these algorithms could be useful in increasing the probability of detecting an event

27. Recommendations
Understand the data- don’t try to make RWD look like an RCT CRF
What you think is “missing” may not be- it just was not needed for patient care
Do studies using RWD because of the clinical utility and scientific value and not driven solely by speed and cost
Doing a pragmatic trial is not a euphemism for “sloppy” or “easy to conduct”.
These trials have a particular place in the spectrum of clinical research methods that continues to evolve
Leverage new technologies to help speed and standardize endpoint “precision” in RWD trials and we are going to be limited by the data.
Transparency and validity in RWD studies are real issues that need to evolve.
This applies both to the data and the algorithms used.

28. Questions and discussion

29. Validation Plan – Source Documents
Source Document Collection
DCRI Coordinating Center will collect source documents via electronic means, fax, and/or mail for all specified endpoints
hospital discharge summaries, brain imaging reports (e.g. CT scans), laboratory values (e.g. hemoglobin levels for bleeding endpoints, cardiac marker values for MI endpoints), and electrocardiograms (for MI endpoints)
These source documents will be collated and private health information and treatment assignment (i.e. aspirin dosing) information will be redacted

30. Validation Plan – Endpoints
Plan to review hospitalizations for (1) myocardial infarction (MI), (2) stroke and (3) major bleeding
Death will not be reviewed since studies have consistently shown that the standard methods to ascertain vital status (i.e. all-cause death) that will be used in ADAPTABLE are valid and accurate
The occurrence of percutaneous coronary intervention (PCI) and coronary artery bypass grafting (CABG) surgery will also not be reviewed since prior studies have shown a high agreement rate for the confirmation of coronary revascularization procedures (κ = 0.88–0.91)

31. ClinicalTrials.gov: NCT02697916
10/2016
ClinicalTrials.gov: NCT