Advanced Statistical Methods for Translational Research Context, Process, Purpose

Disclosure Statement of Financial Interest Within the past 12 months, I or my spouse/partner have had a financial interest/arrangement or affiliation with the organization(s) listed below. Affiliation/Financial Relationship Company My employer provides testing and consulting services to medical device companies Multiple All TCT 2018 faculty disclosures are listed online and on the App.

We are driven to make a scientific contribution to every medical device in the world

Why do we need “advanced” statistical methods? Better answers Faster answers Cheaper answers

Dangers of “advanced” statistical methods? Wrong answers Delayed answers Expensive answers

Dangers of “advanced” statistical methods? Wrong answers Delayed answers Expensive answers “Being a statistician is like being in Alcoholics Anonymous. You try as hard as you can to not fool yourself” Paraphrasing Jim Hodges

Examples of advanced methods Multiple imputation for missing data Propensity score adjustment for non-randomized comparisons Finkelstein-Schoenfeld / “Win-ratio” for composite endpoints Hiearchical models Bayesian approaches: sample size re-estimation, using historical data

How do we evaluate a new statistical method? Does the novel method address a practical problem? What problems does a novel method raise? Role of pre-specification Operating characteristics Urgency of public health need 𝑞(𝜃| 𝐷 0 , 𝐷, 𝛼 0 )∝ 𝐿(𝜃|𝐷)𝐿(𝜃| 𝐷 0 ) 𝛼 0 𝜋 0 (𝜃)

Does the method address a practical problem? If not, why not use a traditional / non-advanced method? How serious is the practical problem? How well does the method address it?

What problems does a novel method raise? Are the assumptions of the method reasonable at the design stage? Are they met for the data we obtained? How thoroughly has the new method been evaluated? Has it been evaluated for the specific circumstances at hand?

Role of pre-specification When we see a new method applied, was it pre-specified? If not, consider the use of the method an exploratory case-study If it was pre-specified, was the study design aligned? Sample size calculations should be based on the planned analysis method Crossover studies, longitudinal studies, etc., require special methods

Operating characteristics Higher power (low type II error rate) Low chance of false-positive (low type I error rate) Under what conditions is the new method optimal/non-optimal?

Urgent public health need Sometimes we need faster answers HIV/AIDS crisis, Ebola vaccine, opioid crisis Device improvements to solve urgent device-related problems Need for high power is the driving consideration What tradeoffs of other considerations can be made? Considerations not purely statistical, e.g. shorter term endpoints

Context, Process, and Purpose P-values… “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold” From: The American Statistical Association Statement on P-values Context, Process, and Purpose

A mock example We have a study with P<0.05!!! Based on a small study Clinically meaningful effect size Analysis based on KM analysis KM stands for Kushner-Manafort test Just kidding – it’s Kaplan-Meier/log-rank test, but the test was not pre-specified

A more realistic example: Kaplan-Meier analysis was once new Addressed practical problems Better handling of time to event data Better handling of censored data (losses-to-follow-up) New problems? Difficulty interpreting? This improved with usage Glosses over issue of informative censoring

Another example: Evaluating methods for “borrowing data” Bayesian methods can be used to “borrow” strength from prior work Formalizes incorporation of existing knowledge Posterior = current data x prior data How much strength should we borrow?

Hierarchical Models

Hierarchical Models Reasonable operating characteristics Only applicable when we have a hierarchical structure Somewhat challenging to describe how much we’re “borrowing”

Prior counts as much as current data Bayesian power prior Traditional Bayesian inference: Nfinal = Ncurrent + Nprior Power prior inference Nfinal = Ncurrent + a*Nprior Prior counts as much as current data

For a <1, prior has less weight Bayesian power prior Traditional Bayesian inference: Nfinal = Ncurrent + Nprior Power prior inference Nfinal = Ncurrent + a*Nprior For a <1, prior has less weight

How should we weight historical data? Historical and current data similar (Easy case) Historical and current data different Weight equally? Ignore prior? Posterior Historical data Current data

Can the weight be dynamic? adaptive? Allow weight to vary as a function of agreement? More agreement = more weight Less agreement = less weight

Advantages of dynamic prior Better type I error rate Less bias

When should we prefer fixed vs. dynamic borrowing? While both approaches rely on theory and data… Fixed borrowing can be considered theory driven We believe a priori that historical and current data should be similar Dynamic borrowing can be considered data driven We allow that historical and current data may or may not be similar

When should we prefer fixed vs. dynamic borrowing? Prefer Theory Driven Prefer Data Driven Reason to believe historical data is representative Questionable if historical data Is representative Identical device, patient population, protocol Small differences in device, patient population, or protocol

Criticisms of dynamic borrowing and responses Data are “used twice”; could introduce bias Shows less bias than other methods of borrowing Other statistical methods use data “twice” (e.g. propensity scores, multiple imputation, interim analyses) Amount of borrowing should depend on study design and clinical considerations Use pre-specification Set specifics of model based on these considerations Lack of straightforward clinical interpretation Criticism also applies to other borrowing methods Common tradeoff with adaptive methods Dynamic borrowing is data driven borrowing http://mdic.org/cts/vp/ MDIC Workshop on Virtual Patient Methodology

For advanced statistical methods, consider… Not just the p-value but the Context, Process, and Purpose

Thank you!