Real World Evidence: Safety An FDA Statistician’s Perspective Estelle Russek-Cohen, PhD

Disclaimer This presentation reflects the views of the author and should not be construed to represent the views or policies of the U.S. Food and Drug Administration. Note: I have been a statistician in each of the three medical centers but each center evolves…..conversations with staff in the three centers has been helpful. www.fda.gov

FDA Three medical product centers but only two broad regulatory paradigms: Center for Drugs Evaluation and Research: Drugs and many biologics Center for Devices and Radiological Devices: Medical Devices Center for Biologics Evaluation and Research: Mostly Biologics, some devices, few drugs This can impact the applicability of FDA guidances.

What is Real World Data Data outside traditional clinical trials (eg): Large simple safety trials Pragmatic clinical trials Registry data (see AHRQ website) Electronic health record data Data from Insurance Claims Common definition of RWD but some data sources are better suited in certain settings.

RWD before 2016 at FDA Commonly used in postmarket safety: Pregnancy registries Product registries (vaccines, devices, drugs, …) ….longer term followup ….capturing rare adverse events Vaccine safety datalink (HMO data) Sentinel (claims/ HMO data)…. mostly drugs & biologics

RWD before 2016 (cont’d) CDRH advanced the use of registries for control arm data using propensity score methods (Yue and colleagues, see refs) FDA used various data sources to study risk benefit. Literature searches and passive surveillance common in postmarket. Real world data can inform clinical trial design (esp. rare diseases)

CDER/CBER Drugs-Biologics Study team expertise Pre-specification Data sources Capture of exposures, confounders, outcomes Design Analysis Sensitivity analysis QA/QC Guidances need to be consistent with rules and laws but they reflect current thinking. https://www.fda.gov/downloads/drugs/guidances/ucm243537.pdf

Device Guidance: RWE CDRH/CBER sponsored Nice glossary Extensive discussion of registries Efficacy and safety discussed. www.fda.gov

Electronic Health Data CDER/CBER/CDRH This guidance for traditional IND/IDEs May have utility for pragmatic trials Challenge of data quality in EHR. ICH posted a GCP reflection paper ….perhaps ICH E6 R2 will be revamped to go beyond randomized trials.

Legislation in 2016 21st Century Cures (FDA and NIH) Reauthorization of User Fees PDUFA VI for drugs and biologics MDUFA IV for medical devices CDER is impacted by PDUFA CDRH impacted by MDUFA CBER impacted by both: depends on product CBER cosigned two adaptive design documents, one for drugs/biologics and one for medical devices.

21st Century Cures Act (2016) establish a program to evaluate the potential use of real world evidence- to help to support the approval of a new indication for a drug approved under section 355(c) of this title; and to help to support or satisfy postapproval study requirements. "real world evidence" means data regarding the usage, or the potential benefits or risks, of a drug derived from sources other than traditional clinical trials. No change in evidentiary standard Pragmatic Trials, Observational Studies 21 Cures directs RWE efforts Post approval studies may be … Include PCT and obs studies —This section shall not be construed to alter— ‘‘(A) the standards of evidence under— ‘‘(i) subsection (c) or (d) of section 505, including the substantial evidence standard in such subsection (d); or SEC. 505F. UTILIZING REAL WORLD EVIDENCE. Amended by Food and Drug Administration Reauthorization Act 2017

Pragmatic Trials A clinical trial designed for the primary purpose of informing decision makers regarding the comparative balance of benefits, burdens and risks of a biomedical or behavioral intervention at the individual or population level. Often hear the term effectiveness rather than efficacy associated with Pragmatic Trials. Pragmatic trials come in different flavors. Some pragmatic trials are cluster randomized. See special issue in Clinical Trials (Sept 2015)

Registries Criteria for eligibility for entire registry: How comprehensive is it? Criteria for eligibility for subjects and data to be used in a study Is the info captured at needed time points using right methods? Agency is likely to provide input at planning stages for design and capturing endpoints of regulatory interest. Talk about ASA Biopharm working group…..discussions suggested that some registries in Europe are tied to government as provider and may be more comprehensive. One would want a credible source maintaining the registry…..American College of Cardiology could be an example. CDRH regularly asked for a postmarket registry to look at long term safety for implanted devices.

Registries versus RCTs Pros Broader inclusion criteria: potential for use in pragmatic trials Same sites as trials: more experience with assessments Cons Length of followup may not be ideal. May need to augment data (eg OS endpoints) In an RCT can have blinding to mitigate some biases

Context for Data: Fit for Purpose? Data captured in the database: Products used in hospital? Use of OTC meds (eg aspirin) Adverse events outside of hospital setting Longitudinal data: Patients move; switch insurers or providers; Multiple visits or adequate longitudinal data on subjects? eg Am I sure this is the first exposure to a vaccine? Timeliness: Claims data can be months to years behind.

Proposing a study using RWE Think carefully about pros and cons Is the RWD fit for current purpose? Reassess? How important is blinding, placebo response? Are patient reported outcomes and clinical outcome assessments of other types important: Are they comparable across raters and studies? Is this appropriately captured in the database? Previous experiences with the product? Can you tell which product is being used? (UDI?) In devices and perhaps in other areas requiring surgery like oncology….how comparable is surgeon skill? Is that appropriately captured in the dataset? For drugs and biologics, In 21st CC emphasis is on data to support additional indications after the first but solid safety data may be considered. Requires discussions with fda. In menb vaccine Boxsero, we allowed additional safety data to go into accelerated approval label…..CDC used the vaccine (already approved in Europe) on college campuses in the US.

Bioethical considerations When is informed consent needed? Randomized trials : always Pragmatic trials: always; registries: ?? Aggregated EHR or claims data: perhaps not How is data being used? Potential for deidentification: Single dataset; Multiple sources combined Registries may involve passive collection of data used only in the aggregate and no changes in treatment decisions and privacy concerns are controlled for. So will depend on the registry.

Big data Bigger is not always better Pros More real world use; Longer term use; Find rare adverse events Cons Potential for confounding is huge Bias: inference only after consider biases Issues common to retrospective analyses: is data recorded at the right time with the right endpoints?

Claims data at FDA Sentinel (www.Sentinel.org) Used initially in postmarket safety Some projects efficacy focused CMS Medicare population: study those >65 Long term followup for those over 65 Data focused on billing….what tests were ordered but results may not be there. Sometimes coding doesn’t identify exact product. Adverse events may not correspond to a code. Lag time while claims are being settled? I remember looking at some vaccination patterns in CMS and they vary by state…..so I am not sure how the implementation of medicare varies by state. One study was done using flu vaccines given in a pharmacy….very different from flu vaccines given in a nursing home environment.

Methodology: Opportunities Better causal inference methods e.g. New methods for propensity scores Randomized studies imbedded in big datasets or registries Methods for distributed data (eg data behind firewalls…) Novel data mining methods Self controlled in vaccines (Kulldorff et al: Tree Scan) Cohort methods in drugs (…) Text mining or data/text mining to utilize info in narratives Methods for combining data sources Methods for studying drug-drug interactions…cannot study every drug combination possible in premarket. Comparative effectiveness (PCORI or FDA?): including network meta-analysis Novel methods for pragmatic trials (incl. cluster randomized trials) Want to mention data mining to develop propensity scores with big data. Many propensity score algorithms use logistic regression which can have issues with lots of possible variables in a model. Less sexy….algorithms to detect problems in the datasets.

Challenges Reeducating clinicians, scientists and statisticians how to work with existing data sources and not always design everything from scratch. Challenge for sponsors and regulators! Incentives for better quality in RWD: don’t assume statistical methods will fix it all. Will insurers be willing to pay for a better understanding of real world outcomes?

Conclusions RWE used in medical product safety before legislation passed in 2016 RWE: bigger role in the future…… possibly used more in efficacy/effectiveness. Pragmatic trials: can be useful to establish real world effectiveness. But limited experience at FDA. Talk with FDA review division before doing study.

References Sources of safety data and statistical strategies for design and analysis: Real world insights. Marchenko et al (2018) TIRS 52: 170-186 Sources of safety data and statistical strategies for Design and analysis: transforming data into evidence. Ma et al (2018) TIRS 52: 187-198 A note on good practice of objective propensity score design for premarket nonrandomized medical device studies with an example. Li et al (2016) SBR: 8: 282-286

References (2) Registries for Evaluating Patient Outcomes: A User’s Guide (3rd edition. Agency for health research quality: available online) Pragmatic Trials. Ford et al 2016 NEJM 375:454-463 Weblink for the ahrq volume is rather long….it is in the cdrh rwe guidance….it is two volumes available for free.