Pragmatic Clinical Trials: Challenges with Clinical Event Ascertainment using RWD Frank W. Rockhold, PhD Professor of Biostatistics and Bioinformatics Duke University Medical Center Duke/Stanford CEC Summit September 26-27, 2018, Chicago, Il

Disclosure Statement – Frank W Rockhold, PhD Research Funding: NIH, PCORI, Duke Clinical Research Institute, Astra Zeneca, ReNeuron, Luitpold, Alzheimer's Drug Discovery Foundation, Janssen, BMS Consulting and Honoraria: https://dcri.org/about-us/conflict-of-interest/ Boards: Frontier Science Board of Directors, DataVant Scientific Advisory Board, European Medicines Agency Technical Advisory Group Equity Interest: GlaxoSmithKline, DataVant

Topics to Discuss Pragmatic Clinical Trials (PCTs) answer questions that are about real world effectiveness. It can be a supplement to and not necessarily a replacement for “classic” (explanatory/confirmatory) RCT’s. Depends on the question and inference desired For this discussion RWD is EMR/Claims data, i.e. data that are a result of a patient health encounter “Investigator” vs GP: Has an impact on recruitment and event ascertainment. Is it really less expensive? In the near term the focus should be on ”how” and not “how much”. Data management vs healthcare informatics- resource and cost shifting. Lessons learned should be of value to all clinical trial conduct. The approach can be useful for safety studies but there needs to be agreement on potential tradeoffs in event ascertainment.

Some questions to consider What loss in ascertainment sensitivity are we willing to live with in a study with RWD study? For safety studies of uncommon or serious events are PCT’s using RWD sources an adequate alternative to RCT’s? OBS study is not likely to fully discharge a safety risk. Do not abandon the concept of randomization too quickly Given practicality considerations what should be our threshold for saying an RWD study is the “only way” even in situations where it is possible to conduct an RCT?? And what is the tradeoff in scientific credibility vs practicality?

PRECIS-2 (Loudon, K. , Treweek, S. , Sullivan, F. , Donnan, P PRECIS-2 (Loudon, K., Treweek, S., Sullivan, F., Donnan, P., Thorpe, K.E., and Zwarenstein, M., The PRECIS-2 tool: designing trials that are fit for purpose. BMJ, 2015. 350: p. h2147 )

Designing a randomised pragmatic clinical trial Ways that Randomised Controlled Trials (RCTs) and PCTs can differ “Classic” RCT P”R”CT Intentionally homogeneous to maximise treatment effect Randomisation and blinding Clinical measures, intermediate endpoints, composite endpoints, clinical outcomes Protocol defines the level and timing of testing. Physicians blinded to data Fixed standard of care or placebo Conducted only by investigators with proven track record Visit schedule and treatment pathway defined in the protocol Patients wishing to change treatment must withdraw from the study Compliance is monitored closely Close monitoring of adherence Intent to treat, per-protocol and completers Eligibility Criteria Randomisation and blinding Endpoints Tests and diagnostics Comparison intervention Practitioner expertise Follow up Continuity Participant compliance Adherence to study protocol Analysis Heterogeneous - representative of normal treatment population Randomisation and rarely blinding Clinical outcomes, PROs, QoL, resource use Measured according to standard practice Standard clinical practice Employment of a variety of practitioners with differing expertise and experience Visits at the discretion of physician and patient. Standard clinical practice – switching therapy according to patient needs Passive monitoring of patient compliance Passive monitoring of practitioner adherence All patients included

ADAPTABLE*, the Aspirin Study – A Patient-Centered Trial *theaspirinstudy.org

PCORnet seeks to improve the nation’s capacity to conduct clinical research by creating a large, highly representative, national patient-centered network that supports more efficient clinical trials and observational studies.

Study Design Patients with known SCVD (ie MI, OR cath ≥75% stenosis of ≥1 epicardial vessel or PCI/CA BG) AND ≥ 1 Enrichment Factor Identified through EMR (computable phenotype) by CDRNs (PPRN pts. already part of a CDRN are eligible) Pts. contacted with trial information and link to eConsent; Treatment assignment provided directly to patient *Enrichment factors Age > 65 years Creatinine > 1.5 Diabetes (Type 1 or 2) 3-vessel coronary artery disease Cerebrovascular disease and/or peripheral artery disease EF <50% by echo, cath, nuclear study Current smoker Exclusion Criteria Age < 18 yrs ASA allergy or contraindication (including pregnancy or nursing) Significant GI bleed within past 12 months Significant bleeding disorder Requires warfarin or NOAC or Ticagrelor ASA 81 mg QD ASA 325 mg QD Electronic F/U Q 3-6 months; Supplemented with EMR/CDM/claims data Duration: Enrollment over 24 months; Maximum f/u of 30 months Primary Endpoint: Composite of all-cause mortality, nonfatal MI, nonfatal stroke Primary Safety Endpoint: Major bleeding complications

ADAPTABLE – How Pragmatic is it?

Centralized Enrollment & Follow-Up Patient Portal Info/eConsent/Randomization Patient Reported Outcomes Medication use Health outcomes Randomized to Q3 or Q6 DCRI Call Center Patients who miss 2 contacts Patient Reported Outcomes Medication use Health outcomes ADAPTABLE Patient 3 6 9 12 15 30 PCORNet Coordinating Center FOLLOW-UP Via Common Data Model Longitudinal health outcomes Baseline Data CMS and private health plans FOLLOW-UP Longitudinal health outcomes

Information Flow Patient PCORnet Supplemental Linkages Mytrus Patient Portal EMR Medicare Claims National Death Index Private Health Plan Data ADAPTABLE Study Database Each data source arrives at the coordinating center via a different mechanism All will contribute to eventual study database Algorithm based decisions for discrepant data/event ascertainment

Information Asymmetry Different participants with different sources of data contributing to endpoint ascertainment Vary by site, Medicare & health plan coverage are not uniform Vary by site or patient if fields are inaccurate or missing PtP EMR CMS NDI HP Patient #1 PtP EMR CMS NDI HP Patient #2 PtP EMR CMS NDI HP Patient #3 PtP EMR CMS NDI HP Patient #4

Information Latency Wg Data Source Availability Min – Max Delay Participant Self-reported data Instant None Electronic Records EMR from PCORnet DataMart Quarterly 3 – 6 months Medicare Claims Data Annual 1 – 12 months National Death Index Private Health Plan Data End of study? Wg Implication: complete data for the study will become available 1 year after the last patient last follow up

Handling Disagreement Across Different Data Sources Patient reported hospitalizations that are not observed in EMR data will be queried via: Medicare fee-for-service claims Large national health plans (FDA’s Mini-Sentinel initiative) DCRI Call Center

Missing Follow-up DCRI Call Center Contact with the site Patient Finder Social Security Death Index

Endpoint Ascertainment and Validation Are we capturing information completely and accurately? A multi-faceted approach to capture outcomes Endpoint validation and Endpoint reconciliation of MACE and Major Bleeding events Can endpoints identified in EMR comparable to clinical endpoints confirmed by traditional adjudication process? Events of interest: MI, stroke, major bleeding Assess agreement: True Positive, False Positive, PPV Provide insights on accuracy of coding algorithms and the data curation process Could a machine algorithm help speed confirmation of events? Absence of events from EMR is not usually verified due to low event rates Could machine algorithms be used to look for potential false negatives?

Approach to Endpoint Ascertainment Routine queries of the PCORnet CDM to capture and classify endpoints using validated coding algorithms Hospitalizations will be identified via standardized, validated coding algorithms developed centrally and applied to the CDM ADAPTABLE patient portal will ask about possible endpoint events (hospitalizations for myocardial infarction, stroke, or major bleeding) during participant contacts (every 3-6 months) Such information will provide additional confirmation for the CDM- generated hospitalization data that meet criteria for the endpoints specified Death ascertainment via Social Security Administration (Medicare beneficiaries) LHC: Red text is not what we describe in the protocol (copied in below). We need to be consistent. Endpoints will be ascertained by applying algorithms designed to ensure comprehensive surveillance for all potential endpoints. •Routine queries will be applied to the PCORnet CDM to capture and classify endpoints using validated coding algorithms. Because the EHR and (by extension) the PCORnet CDM is the source of truth for the trial, no patient confirmation or other additional confirmation will be required. •Hospitalizations for stroke, MI, or bleeding reported by patients will be evaluated in the following manner: oQueries of the PCORnet CDM will be used to cross-check, confirm, and classify in-network hospitalizations (nonfatal MI, nonfatal stroke, or major bleeding) reported by patients. oOut-of-network hospitalizations reported by patients and not captured in the CDM will be identified by patient self-report through the periodic electronic contacts planned throughout the study.

Validation Plan – Background (courtesy of Dr. Schuyler Jones) Formal clinical endpoint classification (CEC) cannot be performed within the budget and will need to be modified in order to ensure efficiency and lower costs Concerns exist that event ascertainment via coding algorithms of administrative claims data needs to demonstrate acceptable levels of validity and be accepted by physicians and patients alike Conflicting data exist regarding the agreement between administrative claims data and clinical study adjudication, thus necessitating the need for this supplemental, validation plan

Validation Plan – Expected Number of Endpoints to Review Random sample of approximately 20% of the MI, stroke, and major bleeding endpoints sampled proportionally from enrolling health systems within participating CDRNs select the first 10 MI events, 10 stroke events, and 5 bleeding events that occur within each CDRN (total=25 events per CDRN). We will randomly select approximately 175 non-fatal MI and stroke endpoints and 50 major bleeding endpoints Approximately 225 overall endpoints to review during the course of the study.

Validation Plan – Adjudication of Events Each case randomly selected will be reviewed in a blinded manner by a disease-specific, expert adjudicator a single cardiologist will review MI and major bleeding endpoints a single neurologist will review stroke endpoints Standard endpoint definitions for MI, stroke, and major bleeding will be used to adjudicate these endpoints using standardized adjudication forms

Validation Plan – Analysis Formal statistical analyses will be performed to measure: percentages of true positive results and false positive results agreement/concordance rates for each endpoint that will be validated (non-fatal stroke, non-fatal MI, and major bleeding) These results (if different) will not change the primary analysis of ADAPTABLE which will count those events identified via coding algorithms as described in the protocol

Data and Safety Monitoring Role of a DMC in a pragmatic trial Standard role Additional focus: feasibility, protocol adherence, data validity Involvement of patient representatives If and how could the DMC make critical decisions with fragmented information during the study? Interim analyses: differentiate a signal from noise with limited access to EMR outcome data Differential data lag times Benefit to risk analysis Could a machine algorithm help even with partial data? Is the DMC protected? Indemnification Source documentation adjudication Concordance analyses

Summary: ADAPTABLE ADAPTABLE is the largest pragmatic trial in the world to evaluate aspirin dose with expected 15,000 participants Attempts to mimic the real world patient experience of a patient with heart disease Collects data through different sources and employs a multi-faceted approach to capture outcomes will teach us a great deal about the value and issues with this approach in the future. Maintained scientific rigor randomised active control robust primary endpoint ADAPTABLE will tell us a great deal about the utility of the approach to perform “mega trials” in a very different way.

Machine Learning for Clinical Events Classification and Safety Surveillance: An Innovation Project from the DCRI ICE-SS group Figure 1.DCRI CEC Process Figure 2. Euclid Trial Summary Statistics 11,082 events Average trigger summary: 46 words (min: 1, max: 1141) Average dossier: 1317 words (min: 1, max: 24984) 80/20/20% partition for training, validation and test, respectively. MI BLEEDING ALI STROKE/TIA DEATH 2088 3966 3019 692 1317 Yes No Stroke Yes TIA Yes NA 853 1235 3044 962 293 2726 317 107 214 Figure 3. Classification Results on Test Set (using trigger summary) The mission of the Duke Clinical Events Classification Group To provide high quality adjudicated endpoint data with scientific rigor, efficiency and innovation by coordinating and conducting Systematic, Comprehensive, Unbiased, Blinded, and Independent clinical events adjudication. Lopes RD, et al. Not published.

Endpoints in Studies (Trial) with RWD PCTs answer questions that are more real world effectiveness. Should be viewed as a supplement to and not a replacement for “classic” RCT’s. “Investigator” vs GP: Has an impact on recruitment and event ascertainment. Is it really less expensive? Time will tell and in short term the focus should be on ”how” and not “how much”. Cost per information unit is unknown. Data management vs healthcare informatics- resource and cost shifting. Lessons learned should be of value to all clinical trial conduct. Machine algorithms could likely help with EMR endpoint PPV. The approach can be useful for safety studies but there needs to be agreement on tradeoffs in event ascertainment and effort involved in data acquisition Even with tradeoffs of potential data gaps these algorithms could be useful in increasing the probability of detecting an event

Recommendations Understand the data- don’t try to make RWD look like an RCT CRF What you think is “missing” may not be- it just was not needed for patient care Do studies using RWD because of the clinical utility and scientific value and not driven solely by speed and cost Doing a pragmatic trial is not a euphemism for “sloppy” or “easy to conduct”. These trials have a particular place in the spectrum of clinical research methods that continues to evolve Leverage new technologies to help speed and standardize endpoint “precision” in RWD trials and we are going to be limited by the data. Transparency and validity in RWD studies are real issues that need to evolve. This applies both to the data and the algorithms used.

Questions and discussion

Validation Plan – Source Documents Source Document Collection DCRI Coordinating Center will collect source documents via electronic means, fax, and/or mail for all specified endpoints hospital discharge summaries, brain imaging reports (e.g. CT scans), laboratory values (e.g. hemoglobin levels for bleeding endpoints, cardiac marker values for MI endpoints), and electrocardiograms (for MI endpoints) These source documents will be collated and private health information and treatment assignment (i.e. aspirin dosing) information will be redacted

Validation Plan – Endpoints Plan to review hospitalizations for (1) myocardial infarction (MI), (2) stroke and (3) major bleeding Death will not be reviewed since studies have consistently shown that the standard methods to ascertain vital status (i.e. all-cause death) that will be used in ADAPTABLE are valid and accurate The occurrence of percutaneous coronary intervention (PCI) and coronary artery bypass grafting (CABG) surgery will also not be reviewed since prior studies have shown a high agreement rate for the confirmation of coronary revascularization procedures (κ = 0.88–0.91)

ClinicalTrials.gov: NCT02697916 10/2016 ClinicalTrials.gov: NCT02697916