Presentation is loading. Please wait.

Presentation is loading. Please wait.

How do we improve the “sufficiency” of the FDA’s Sentinel System?

Published byCharla Palmer Modified over 6 years ago

Similar presentations

Presentation on theme: "How do we improve the “sufficiency” of the FDA’s Sentinel System?"— Presentation transcript:

1 How do we improve the “sufficiency” of the FDA’s Sentinel System?
Robert Ball, MD, MPH, ScM Deputy Director Office of Surveillance and Epidemiology Center of Drug Evaluation and Research March 28, 2017

2 2007 FDA Amendments Act (FDAAA)
Post Marketing Requirements Safety Labeling Changes Risk Evaluation and Mitigation Strategies (REMS) Required Safety Reviews (“915” and “921”) Active post-market Risk Identification and Analysis system (ARIA) FDA Sentinel Initiative

3 Sentinel System Now Activated
Transition Year Standup a more formal process for ARIA system to meet FDAAA 2007 mandate Mini-Sentinel Pilot 5-year pilot to establish the methods and distributed database architecture to conduct safety surveillance 2009 2010 2011 2012 2013 2014 2015 2016 Activated Sentinel System Formally implement ARIA and capture reasons for insufficiency

4 Mini-Sentinel Accomplishments
Mini-Sentinel Coordinating Center > 30 Collaborating institutions (18 Data Partners) Common Data Model & Distributed data network Secure querying behind data partners’ firewall Access to quality checked, electronic healthcare data of over 178 million patient Pilot to develop the scientific and technical operations for an active medical product safety surveillance system

5 Summary Description Sentinel System Characteristics
193 million individuals in 18 data partners Access to laboratory, pharmacy and medical records Primarily insured population Adding Medicare and a large hospital network with EMR Distributed system ensures privacy and security Data not pooled into single database Data partners retain physical control Analytic programs run against common data model

6 Pros and Cons of Claims Data
Advantages Disadvantages Capture all reimbursed healthcare usage Data are longitudinal and can produce incidence rates Use for payment contributes to data quality Common billing standards allow aggregation into large sample sizes Data are not collected specifically for research Economic incentives affect data Missing OTC, low-cost generics paid out of pocket, drug samples, etc. Challenging to get key lifestyle factors (e.g., smoking, diet, exercise)

7 Active Risk Identification and Analysis (ARIA) System
Mandated creation in Section 905 of FDAAA 2007 Linked to PMR in Section 901(3)(D)(i): “The Secretary may not require the responsible person to conduct a study under this paragraph, unless the Secretary makes a determination that the reports under subsection (k)(1) and the active postmarket risk identification and analysis system as available under subsection (k)(3) will not be sufficient to meet the purposes set forth in subparagraph (B).”

8 Defining ARIA ARIA uses a subset of Sentinel System’s full capabilities to fulfill the FDAAA mandate to conduct active safety surveillance Analytic Tools* Common Data Model† ARIA * Pre-defined, parameterized, and re-usable to enable faster safety surveillance in Sentinel (in contrast to protocol based assessments with customized programming) † Electronic claims data, without manual medical record review

9 Level 2 Adjusted Analyses with Sophisticated Confounding Control
ARIA is Comprised of Distributed Querying Approach using Modular Programs Level 3 Sequential Adjusted Analyses with Sophisticated Confounding Control Level 2 Adjusted Analyses with Sophisticated Confounding Control Level 1 Descriptive Analyses, Unadjusted Rates Modular Programs Currently in ARIA Future ARIA Capabilities

10 Sentinel Surveillance Toolbox

11 Common Data Model

12 What is ARIA Sufficiency?
Adequate data Drug of interest and comparator Health outcome of interest Confounders and covariates Appropriate methods To answer the question of interest assess a known serious risk related to the use of the drug assess signals of serious risk related to the use of the drug identify an unexpected serious risk when available data indicate the potential for a serious risk To lead to a satisfactory level of precision

13 When is ARIA Insufficient?
Inadequate data Health outcome of interest – claims alone not always adequate (e.g. anaphylaxis) Creating cohorts – variety of reasons (e.g. cancer and missing data) Confounders and covariates – not always available in common data model (e.g. BMI, smoking history)

14 EHR narratives vs Coded data
Key Points “Text in EMRs is accessible, especially with open source information extraction algorithms, and significantly improves case detection when combined with codes. More harmonization of reporting within EMR studies is needed, particularly standardized reporting of algorithm accuracy metrics like positive predictive value (precision) and sensitivity (recall).” Authors also noted small sample that directly compared codes to narratives and variability in performance. Ford E et al. J Am Med Inform Assoc 23 (5),

15 Health Outcome of Interest: Anaphylaxis
KEY POINTS • The authors developed and validated an algorithm using administrative and claims data to identify cases of anaphylaxis. • The PPV for the overall algorithm was 63.1% (95% CI: %). While this PPV improves on previous publications, it remains low. • The authors were able to identify an algorithm that optimized the PPV but demonstrated lower sensitivity for anaphylactic events. Walsh KE et al. Validation of anaphylaxis in the Food and Drug Administration’s Mini-Sentinel. Pharmacoepidemiology and drug safety 2013; 22: 1205–1213

16 Health Outcome of Interest: Anaphylaxis
KEY POINTS • The authors developed an algorithm to extract key features from narratives of Vaccine Adverse Event Report System (VAERS) reports using natural language processing. • The authors used those features to classify reports of possible anaphylaxis after vaccination based on the Brighton Collaboration definition using both a rule-based and similarity-based classifier. Botsis T, et al. Vaccine Adverse Event Text Mining (VaeTM) system for extracting features from vaccine safety reports. J Am Med Inform Assoc 19: , 2012.

17 Health Outcome of Interest: Anaphylaxis
KEY POINTS • The previously developed natural language processing, rule- and similarity-based classification approaches demonstrated almost equal performance (F-measure: vs , recall 100% vs 100%, precision 60.3% vs 57.4%). • These algorithms might improve recall but had similar precision (PPV) to claims only algorithms from MS. Ball et al, Evaluating automated approaches to anaphylaxis case classification using unstructured data from the FDA Sentinel System, in preparation

18 Health Outcome of Interest: Anaphylaxis
KEY POINTS • Reasons for misclassification included: the inability of the algorithms to make the same clinical judgments as human experts about the timing, severity, or presence of alternative explanations; the identification of terms consistent with anaphylaxis but present in conditions other than anaphylaxis. Ball et al, Evaluating automated approaches to anaphylaxis case classification using unstructured data from the FDA Sentinel System, in preparation

19

20 Challenges to Assessing Cancer as a Drug Safety Outcome
Data Sources Methods Existing databases have large sample sizes for timely, cost-effective analyses Continuity of coverage limited to 2-5 years Cancer registries high quality data, but require linkage to collect the needed non-cancer information Algorithms have low PPV for many cancers Unknown risk window and latency Risk estimation needs to accommodate time varying nature of cancer risk

21 http://www. nejm. org/doi/full/10. 1056/NEJMp1606181

22 Promise of Electronic Health Records
“ high-value EHR data are often stored in unstructured formats that are inaccessible to algorithms without layers of preprocessing… models need to be built and validated individually for each diagnosis… Obermeyer Z, Emanuel EJ. Predicting the Future — Big Data, Machine Learning, and Clinical Medicine. N Engl J Med 2016; 375:

23 Summary The Sentinel System (and ARIA), was designed to be a scalable, rapidly responsive analytic system for all FDA regulated medical products Sentinel offers numerous potential uses From drug use to sophisticated pharmacoepidemiology analyses Principal challenges are limitations of claims data and their “sufficiency” for the many safety concerns at FDA Great opportunities exist with EHR, if we can figure out how to harness the data

24 Acknowledgments Michael Nguyen MD, FDA Sentinel Program Lead
Taxiarchis Botsis PhD, CBER, FDA Jeff Brown PhD, Sentinel Operations Center

25 Thank you

Download ppt "How do we improve the “sufficiency” of the FDA’s Sentinel System?"

Similar presentations

Welcome to Game Lets start the Game. An electronic health record (EHR) is a digital version of a patient’s paper chart. EHRs are real-time, patient-centered.

Welcome to Game Lets start the Game. An electronic health record (EHR) is a digital version of a patient’s paper chart. EHRs are real-time, patient-centered.
CHAPTER © 2011 The McGraw-Hill Companies, Inc. All rights reserved. 2 The Use of Health Information Technology in Physician Practices.

CHAPTER © 2011 The McGraw-Hill Companies, Inc. All rights reserved. 2 The Use of Health Information Technology in Physician Practices.
1 Proposed Pharmacovigilance Plan for H5N1 Influenza Virus Vaccine Patrick Caubel, MD, PhD Head of Pharmacovigilance North America February 27, 2007.

1 Proposed Pharmacovigilance Plan for H5N1 Influenza Virus Vaccine Patrick Caubel, MD, PhD Head of Pharmacovigilance North America February 27, 2007.
Potential Roles and Limitations of Biomarkers in Alzheimer’s Disease Richard Mayeux, MD, MSc Columbia University.

Potential Roles and Limitations of Biomarkers in Alzheimer’s Disease Richard Mayeux, MD, MSc Columbia University.
ClinicalTrials.gov Stephen Kiyoi

ClinicalTrials.gov Stephen Kiyoi
Meaningful Use, Standards and Certification Under HITECH—Implications for Public Health InfoLinks Community of Practice January 14, 2010 Bill Brand, MPH,

Meaningful Use, Standards and Certification Under HITECH—Implications for Public Health InfoLinks Community of Practice January 14, 2010 Bill Brand, MPH,
Surveillance. Definition Continuous and systematic process of collection, analysis, interpretation, and dissemination of descriptive information for monitoring.

Surveillance. Definition Continuous and systematic process of collection, analysis, interpretation, and dissemination of descriptive information for monitoring.
Selection of Data Sources for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ) www.ahrq.gov.

Selection of Data Sources for Observational Comparative Effectiveness Research Prepared for: Agency for Healthcare Research and Quality (AHRQ)
Cohort Studies Hanna E. Bloomfield, MD, MPH Professor of Medicine Associate Chief of Staff, Research Minneapolis VA Medical Center.

Cohort Studies Hanna E. Bloomfield, MD, MPH Professor of Medicine Associate Chief of Staff, Research Minneapolis VA Medical Center.
Postmarketing Risk Assessment of Drug Products Division of Drug Risk Evaluation Office of Drug Safety Center for Drug Evaluation and Research.

Postmarketing Risk Assessment of Drug Products Division of Drug Risk Evaluation Office of Drug Safety Center for Drug Evaluation and Research.
Unique Device Identification and Global Medical Device Nomenclature Jay Crowley Senior Advisor for Patient Safety Food and Drug Administration

Unique Device Identification and Global Medical Device Nomenclature Jay Crowley Senior Advisor for Patient Safety Food and Drug Administration
FDA Recalls Risk Communication Advisory Committee David K. Elder Director, Office of Enforcement.

FDA Recalls Risk Communication Advisory Committee David K. Elder Director, Office of Enforcement.
Patient Safety and Public Health Informatics Iona Thraen, ACSW Patient Safety Director.

Patient Safety and Public Health Informatics Iona Thraen, ACSW Patient Safety Director.
DSaRM Advisory Committee May 18, 2005 Active Surveillance for Drug Safety Signals: Past, Present, and Future Mary Willy, Ph.D. Division of Drug Risk Evaluation.

DSaRM Advisory Committee May 18, 2005 Active Surveillance for Drug Safety Signals: Past, Present, and Future Mary Willy, Ph.D. Division of Drug Risk Evaluation.
Research Techniques Made Simple: Databases for Clinical Research Katrina Abuabara, MD MA David Margolis, MD PhD University of Pennsylvania.

Research Techniques Made Simple: Databases for Clinical Research Katrina Abuabara, MD MA David Margolis, MD PhD University of Pennsylvania.
The Use of Health Information Technology in Physician Practices

The Use of Health Information Technology in Physician Practices
SALUS - Scalable, Standard based Interoperability Framework for Sustainable Proactive Post Market Safety Studies A short overview May 2015 A. Anil SINACI.

SALUS - Scalable, Standard based Interoperability Framework for Sustainable Proactive Post Market Safety Studies A short overview May 2015 A. Anil SINACI.
Postmarketing Safety Assessment of Osteonecrosis of the Jaw Pamidronate & Zoledronic Acid Division of Drug Risk Evaluation Office of Drug Safety FDA Carol.

Postmarketing Safety Assessment of Osteonecrosis of the Jaw Pamidronate & Zoledronic Acid Division of Drug Risk Evaluation Office of Drug Safety FDA Carol.
LEADING RESEARCH… MEASURES THAT COUNT Challenges of Studying Cardiovascular Outcomes in ADHD Elizabeth B. Andrews, MPH, PhD, VP, Pharmacoepidemiology and.

LEADING RESEARCH… MEASURES THAT COUNT Challenges of Studying Cardiovascular Outcomes in ADHD Elizabeth B. Andrews, MPH, PhD, VP, Pharmacoepidemiology and.
The HMO Research Network (HMORN) is a well established alliance of 18 research departments in the United States and Israel. Since 1994, the HMORN has conducted.

The HMO Research Network (HMORN) is a well established alliance of 18 research departments in the United States and Israel. Since 1994, the HMORN has conducted.

Similar presentations

Ads by Google