1 U sing Disproportionality Analysis to Systematically and Simultaneously Detect Safety Signals in AERS Jonathan G. Levine, PhD FDA/CDER Office Of Pharmacoepidemiology & Statistical Science

2 Disclaimer The views expressed in this presentation are mine, and do not necessarily represent the views of FDA.

3 Overview Goals of adverse event analysis. What is AERS? Limitations of AERS AERS versus Clinical Trials. What can we learn from AERS? Disproportionality Analysis. Interpreting Disproportionality Scores

4 Goals of Adverse Event Analysis Adverse event analysis seeks to answer these three questions: –Which drugs or combinations of drugs cause which adverse events? –Which patients are most likely to experience the adverse event? –Is an alternate therapy more or less likely to cause the event? AERS data cannot by itself answer these questions AERS data can help to answer these questions

5 AERS Database Computerized adverse case reporting system –Voluntary reporting by health care workers and the general public. –Mandatory reporting by manufacturers for serious, unexpected events Adverse event reports –Coded according to the standardized terminology of the Medical Dictionary for Regulatory Activities(MedDRA) –Over 2.5 million voluntary reports from 1968 to the present. –Small number of data elements (drugs, events, age, sex, etc.) –Lots of missing data Safety issues leading to drug withdrawal from the market have been discovered using the AERS database

6 Limitations of AERS No denominator –Population rates cannot be estimated. Differential under-reporting of events can bias results. –Consistent under-reporting for a drug or event does not bias results –Under-reporting that is a function of drug and event does bias results Drug name errors –Data entry is not done following strict drug naming standards. Missing data MedDRA is often too granular –Numerous terms for seizures, strokes, etc. MedDRA codes are often difficult to aggregate –Higher level groupings may contain an event and its opposite, e.g. hypertension and hypotension. MedDRA is not comprehensive –A very unusual adverse event may not have a MedDRA term. Duplicate reports –Multiple manufacturers may submit the same report.

7 AERS versus Clinical Trials Representative population? –Clinical trials: non-random samples, certain types of patients are intentionally excluded. –AERS: non-random sample, no one intentionally excluded. Generalizability –Clinical trials: results may not apply to other patient populations. –AERS: results apply to the entire patient population (in theory); reporting may be driven by events incompletely addressed in the labeling. Number of patients studied –Clinical trials: small (hundreds, maybe thousands of patients). –AERS: everyone (in theory…but under-reporting biases and labeling of an event may in practice reduce the number of patients effectively studied).

8 AERS versus Clinical Trials Ascertainment? –Clinical trials: good ascertainment for serious events with fairly rapid onset and more subtle events that are the focus of monitoring –AERS: under-reporting Causality –Clinical trials: Randomization makes causality assessment straightforward. –AERS: causality assessment difficult without using external information. ( e.g. good patient descriptions, linked to an objective marker such as digoxin levels, potassium levels, and a correlated EKG and positive dechallenges and rechallenges) Complexity of research question –Clinical trials: limited by cost and ethics. –AERS: limited by actual drug use and reporting.

9 What can a statistician learn from AERS? Has adverse event Y been reported in patients taking drug X? Are reports in AERS containing drug X more likely to also contain adverse event Y? Question: How can we analyze AERS data in order to answer these questions?

10 Disproportionality Analysis Using DuMouchels MGPS Method Calculate observed and expected number of reports for a particular drug- event combination. Observed rate = Number of reports for event X with drug Y Number of reports for drug Y Expected rate = Number of reports for event X in AERS Number of reports in AERS Reporting Ratio (RR)= Observed rate Expected rate Shrink the RR towards 1. The shrunk RR is referred to as the EBGM score. The amount of shrinkage is a function of the amount of information in AERS about the drug-event combination.

11 Disproportionality Analysis MGPS done on a cleaned version of AERS –Duplicate removal, drug name standardization All scores are calculated simultaneously. Expected rates are usually calculated stratifying by age, sex, and report year. The strata-level expected rates are combined using strata weights proportional to strata size. A common measure of the importance of a drug- event combination is the lower limit of the 90% credible region (confidence interval) for the EBGM, referred to as EB05.

12 What Does an EB05>1 Mean? An EB05>1 suggests that in AERS there is an association between the drug and the event. An EB05>1 does not prove that a drug causes an event. What is a large EB05 value? –A signal having an EB05 > 2 indicates that the D-E is at least twice the expected –In many cases, an EB05 >1, or EB05 >1.5 may be a more useful definition.

13 When Are Findings from Disproportionality Analysis Most Compelling? Very low background rate in general population, e.g. growing feathers Very low background rate in similar patients Objective outcome Any finding must be viewed in context.

14 A. B. Hill not R. A. Fisher ( Hill asked: What aspects of that association should we especially consider before deciding that the most likely interpretation of it is causation? –Strength. –Consistency –Specificity –Temporality –Biological gradient –Plausibility –Coherence –Experiment –Analogy Determination of causality requires more than statistics.

15 Does Disproportionality Analysis Work? MGPS has identified emerging adverse events. EB05s > 1 are usually seen for known adverse events. Implausible signals are rarely seen, (but sometimes the event causes the drug; need medical input to determine direction of causality). Innocent bystanders problem and signal leakage in cases of polypharmacy with independence model are still a problem. As a means to address these issues, Bayesian logistic regression methods are being studied.

16 Conclusions AERS provides useful information about adverse events. Clinical trials cannot replace the information provided by AERS. Biases exist in AERS, and the exact nature of the biases is impossible to determine. Disproportionality analysis can provide an understanding of the associations between drug-event pairs in AERS. Disproportionality analysis of AERS cannot by itself determine if there is a causal link between a drug-event pair.

17 Selected References Almenoff, J. S., W. DuMouchel, A. Kindman, X. Yang and D. M. Fram (2003). "Disproportionality analysis using empirical Bayes data mining: a tool for the evaluation of drug interactions in the post-marketing setting." Pharmacoepidemiology and Drug Safety 12(6): DuMouchel, W., Bayesian data mining in large frequency tables, with an application to the FDA Spontaneous Reporting System. The American Statistician, (3): DuMouchel, W. and D. Pregibon, Empirical Bayes screening for multi-item associations. In 7th ACM SigKDD Intl Conference on Knowledge Discovery and Data Mining San Francisco: ACM Press. Fram DM, Almenoff JS, DuMouchel W (2003) Empirical Bayesian Data Mining for Discovering Patterns in Post-Marketing Drug Safety Data Proc. ACMSIGKDD 2003 Intl. Conf. on Knowledge Discovery from Data. Niu, M.T., D.E. Erwin, and M.M. Braun, Data mining in the US Vaccine Adverse Event Reporting System (VAERS): early detection of intussusception and other events after rotavirus vaccination. Vaccine, : O'Neill, R.T. and A. Szarfman, Discussion: Bayesian data mining in large frequency tables, with an application to the FDA Spontaneous Reporting System by William DuMouchel. The American Statistician, (3): O'Neill, R.T. and A. Szarfman, Some FDA perspectives on data mining for pediatric safety assessment. Curr Ther Res Clin Exp, : Szarfman, A., S.G. Machado, and R.T. ONeill. Use of Screening Algorithms and Computer Systems to Efficiently Signal Higher-Than-Expected Combinations of Drugs and Events in the US FDAs Spontaneous Reports Database. Drug Safety, (6): p van Puijenbroek EP, Diemont WL, van Grootheest K Application of Quantitative Signal Detection in the Dutch Spontaneous Reporting System for Adverse Drug Reactions Drug Safety 2003; 26 (5):