How do we improve the “sufficiency” of the FDA’s Sentinel System? Robert Ball, MD, MPH, ScM Deputy Director Office of Surveillance and Epidemiology Center of Drug Evaluation and Research March 28, 2017

2007 FDA Amendments Act (FDAAA) Post Marketing Requirements Safety Labeling Changes Risk Evaluation and Mitigation Strategies (REMS) Required Safety Reviews (“915” and “921”) Active post-market Risk Identification and Analysis system (ARIA) FDA Sentinel Initiative

Sentinel System Now Activated Transition Year Standup a more formal process for ARIA system to meet FDAAA 2007 mandate Mini-Sentinel Pilot 5-year pilot to establish the methods and distributed database architecture to conduct safety surveillance 2009 2010 2011 2012 2013 2014 2015 2016 Activated Sentinel System Formally implement ARIA and capture reasons for insufficiency

Mini-Sentinel Accomplishments Mini-Sentinel Coordinating Center > 30 Collaborating institutions (18 Data Partners) Common Data Model & Distributed data network Secure querying behind data partners’ firewall Access to quality checked, electronic healthcare data of over 178 million patient Pilot to develop the scientific and technical operations for an active medical product safety surveillance system

Summary Description Sentinel System Characteristics 193 million individuals in 18 data partners Access to laboratory, pharmacy and medical records Primarily insured population Adding Medicare and a large hospital network with EMR Distributed system ensures privacy and security Data not pooled into single database Data partners retain physical control Analytic programs run against common data model https://www.sentinelsystem.org/

Pros and Cons of Claims Data Advantages Disadvantages Capture all reimbursed healthcare usage Data are longitudinal and can produce incidence rates Use for payment contributes to data quality Common billing standards allow aggregation into large sample sizes Data are not collected specifically for research Economic incentives affect data Missing OTC, low-cost generics paid out of pocket, drug samples, etc. Challenging to get key lifestyle factors (e.g., smoking, diet, exercise)

Active Risk Identification and Analysis (ARIA) System Mandated creation in Section 905 of FDAAA 2007 Linked to PMR in Section 901(3)(D)(i): “The Secretary may not require the responsible person to conduct a study under this paragraph, unless the Secretary makes a determination that the reports under subsection (k)(1) and the active postmarket risk identification and analysis system as available under subsection (k)(3) will not be sufficient to meet the purposes set forth in subparagraph (B).” https://www.gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf

Defining ARIA ARIA uses a subset of Sentinel System’s full capabilities to fulfill the FDAAA mandate to conduct active safety surveillance Analytic Tools* Common Data Model† ARIA * Pre-defined, parameterized, and re-usable to enable faster safety surveillance in Sentinel (in contrast to protocol based assessments with customized programming) † Electronic claims data, without manual medical record review

Level 2 Adjusted Analyses with Sophisticated Confounding Control ARIA is Comprised of Distributed Querying Approach using Modular Programs Level 3 Sequential Adjusted Analyses with Sophisticated Confounding Control Level 2 Adjusted Analyses with Sophisticated Confounding Control Level 1 Descriptive Analyses, Unadjusted Rates Modular Programs Currently in ARIA Future ARIA Capabilities

Sentinel Surveillance Toolbox

Common Data Model https://www.sentinelsystem.org/sentinel/data/distributed-database-common-data-model

What is ARIA Sufficiency? Adequate data Drug of interest and comparator Health outcome of interest Confounders and covariates Appropriate methods To answer the question of interest assess a known serious risk related to the use of the drug assess signals of serious risk related to the use of the drug identify an unexpected serious risk when available data indicate the potential for a serious risk To lead to a satisfactory level of precision

When is ARIA Insufficient? Inadequate data Health outcome of interest – claims alone not always adequate (e.g. anaphylaxis) Creating cohorts – variety of reasons (e.g. cancer and missing data) Confounders and covariates – not always available in common data model (e.g. BMI, smoking history)

EHR narratives vs Coded data Key Points “Text in EMRs is accessible, especially with open source information extraction algorithms, and significantly improves case detection when combined with codes. More harmonization of reporting within EMR studies is needed, particularly standardized reporting of algorithm accuracy metrics like positive predictive value (precision) and sensitivity (recall).” Authors also noted small sample that directly compared codes to narratives and variability in performance. Ford E et al. J Am Med Inform Assoc 23 (5), 1007-1015. 2016.

Health Outcome of Interest: Anaphylaxis KEY POINTS • The authors developed and validated an algorithm using administrative and claims data to identify cases of anaphylaxis. • The PPV for the overall algorithm was 63.1% (95% CI: 53.9-71.7%). While this PPV improves on previous publications, it remains low. • The authors were able to identify an algorithm that optimized the PPV but demonstrated lower sensitivity for anaphylactic events. Walsh KE et al. Validation of anaphylaxis in the Food and Drug Administration’s Mini-Sentinel. Pharmacoepidemiology and drug safety 2013; 22: 1205–1213

Health Outcome of Interest: Anaphylaxis KEY POINTS • The authors developed an algorithm to extract key features from narratives of Vaccine Adverse Event Report System (VAERS) reports using natural language processing. • The authors used those features to classify reports of possible anaphylaxis after vaccination based on the Brighton Collaboration definition using both a rule-based and similarity-based classifier. Botsis T, et al. Vaccine Adverse Event Text Mining (VaeTM) system for extracting features from vaccine safety reports. J Am Med Inform Assoc 19:1011-1018, 2012.

Health Outcome of Interest: Anaphylaxis KEY POINTS • The previously developed natural language processing, rule- and similarity-based classification approaches demonstrated almost equal performance (F-measure: 0.753 vs. 0.729, recall 100% vs 100%, precision 60.3% vs 57.4%). • These algorithms might improve recall but had similar precision (PPV) to claims only algorithms from MS. Ball et al, Evaluating automated approaches to anaphylaxis case classification using unstructured data from the FDA Sentinel System, in preparation

Health Outcome of Interest: Anaphylaxis KEY POINTS • Reasons for misclassification included: the inability of the algorithms to make the same clinical judgments as human experts about the timing, severity, or presence of alternative explanations; the identification of terms consistent with anaphylaxis but present in conditions other than anaphylaxis. Ball et al, Evaluating automated approaches to anaphylaxis case classification using unstructured data from the FDA Sentinel System, in preparation

http://www.sciencedirect.com/science/article/pii/S1047279716302769

Challenges to Assessing Cancer as a Drug Safety Outcome Data Sources Methods Existing databases have large sample sizes for timely, cost-effective analyses Continuity of coverage limited to 2-5 years Cancer registries high quality data, but require linkage to collect the needed non-cancer information Algorithms have low PPV for many cancers Unknown risk window and latency Risk estimation needs to accommodate time varying nature of cancer risk

http://www. nejm. org/doi/full/10. 1056/NEJMp1606181 http://www.nejm.org/doi/full/10.1056/NEJMp1606181?query=featured_home

Promise of Electronic Health Records “ high-value EHR data are often stored in unstructured formats that are inaccessible to algorithms without layers of preprocessing… models need to be built and validated individually for each diagnosis… Obermeyer Z, Emanuel EJ. Predicting the Future — Big Data, Machine Learning, and Clinical Medicine. N Engl J Med 2016; 375:1216-1219

Summary The Sentinel System (and ARIA), was designed to be a scalable, rapidly responsive analytic system for all FDA regulated medical products Sentinel offers numerous potential uses From drug use to sophisticated pharmacoepidemiology analyses Principal challenges are limitations of claims data and their “sufficiency” for the many safety concerns at FDA Great opportunities exist with EHR, if we can figure out how to harness the data

Acknowledgments Michael Nguyen MD, FDA Sentinel Program Lead Taxiarchis Botsis PhD, CBER, FDA Jeff Brown PhD, Sentinel Operations Center

Thank you