A Vision for Clinical Trials Scott Evans, MS, PhD Harvard University May, 2017

Significant Contributors (p<0.001) Dean Follmann Dan Rubin Chip Chambers Thuy Tran Judith Lok Michelle Earley David van Duin

Most clinical trials fail to provide the evidence needed to inform medical decision-making. However, the serious implications of this deficit are largely absent from public discourse. DeMets and Califf, JAMA, 2011

The medical community is calling for more: Systematic evaluation of benefits and harms Pragmatism

If I had one hour to solve a problem, I would spend 55 minutes defining and understanding the problem, and 5 minutes solving it. Albert Einstein

Question 1 Suppose we measure the duration of hospitalization Shorter duration is better … or is it? The faster the patient dies, the shorter the duration Interpretation of an outcome need clinical context of other outcomes for the same patient

Question 2 We define analysis populations Efficacy analysis: efficacy population (e.g., ITT) Safety analysis: safety population Efficacy population ≠ safety population We combine these analyses into benefit:risk analyses To whom does this analysis apply?

Question 3 Suppose a loved one is diagnosed with a serious disease You are selecting treatment 3 treatment options: A, B, and C 2 outcomes Treatment success: yes/no Safety event: yes/no

RCT Comparing A, B, and C Analysis of Outcomes B (N=100) C (N=100)

RCT Comparing A, B, and C Analysis of Outcomes Success: 50% B (N=100) Success: 50% C (N=100) Success: 50%

RCT Comparing A, B, and C Analysis of Outcomes Success: 50% Safety event: 30% B (N=100) Success: 50% Safety event: 50% C (N=100) Success: 50% Safety event: 50%

RCT Comparing A, B, and C Analysis of Outcomes Success: 50% Safety event: 30% B (N=100) Success: 50% Safety event: 50% C (N=100) Success: 50% Safety event: 50% Which treatment would you choose?

RCT Comparing A, B, and C Analysis of Outcomes Success: 50% Safety event: 30% B (N=100) Success: 50% Safety event: 50% C (N=100) Success: 50% Safety event: 50% Which treatment would you choose? They all have the same success rate.

RCT Comparing A, B, and C Analysis of Outcomes Success: 50% Safety event: 30% B (N=100) Success: 50% Safety event: 50% C (N=100) Success: 50% Safety event: 50% Which treatment would you choose? They all have the same success rate. A has the lowest safety event rate.

RCT Comparing A, B, and C Analysis of Outcomes Success: 50% Safety event: 30% B (N=100) Success: 50% Safety event: 50% C (N=100) Success: 50% Safety event: 50% Which treatment would you choose? They all have the same success rate. A has the lowest safety event rate. B and C are indistinguishable.

RCT Comparing A, B, and C Analysis of Outcomes Success: 50% Safety event: 30% B (N=100) Success: 50% Safety event: 50% C (N=100) Success: 50% Safety event: 50% Which treatment would you choose? They all have the same success rate. A has the lowest safety event rate. B and C are indistinguishable. Choose A…right?

Analysis of Patients: 4 Possible Outcomes Success: 50% Safety event: 30% B (N=100) Success: 50% Safety event: 50% C (N=100) Success: 50% Safety event: 50% Success + - Success + - Success + - SE + - 15 35 50 50

Analysis of Patients: 4 Possible Outcomes Success: 50% Safety event: 30% B (N=100) Success: 50% Safety event: 50% C (N=100) Success: 50% Safety event: 50% Success + - Success + - Success + - SE + - 15 35 50 50

Analysis of Patients: 4 Possible Outcomes Success: 50% Safety event: 30% B (N=100) Success: 50% Safety event: 50% C (N=100) Success: 50% Safety event: 50% Success + - Success + - Success + - SE + - 15 35 50 50

Analysis of Patients: 4 Possible Outcomes Success: 50% Safety event: 30% B (N=100) Success: 50% Safety event: 50% C (N=100) Success: 50% Safety event: 50% Success + - Success + - Success + - SE + - 15 35 50 50

Our culture is to use patients to analyze the outcomes.

Our culture is to use patients to analyze the outcomes Our culture is to use patients to analyze the outcomes. Shouldn’t we use outcomes to analyze the patients?

Scott’s father (a math teacher) to his confused son many years ago: “The order of operations is important…”

Instead of getting an optimal solution to the wrong problem, let’s get A solution to the right problem.

Translational Statistics Conventional analysis provides fragmented information Lack of synthesis of the totality of evidence / total disease burden Difficult to use trial results to inform patient management Let’s design and analyze studies to have greater applicability for clinical practice

A Vision

The Future of Clinical Trials Today Tomorrow Endpoints Many Global patient outcome Patient / Clinician Preferences Limited Incorporated Treatment Effects One Many (personalized)

The Future of Clinical Trials Today Tomorrow Endpoints Many Global patient outcome Patient / Clinician Preferences Limited Incorporated Treatment Effects One Many (personalized)

Pairwise Comparison of Patients Rank patients based on a global patient outcomes synthesizing benefits and harms Estimate Win ratio: # wins / # losses DOOR probability: the probability of a more desirable global outcome when assigned to the new vs. the standard treatment

Desirability of Outcome Ranking (DOOR) Before we analyze several hundred patients, we must understand how to analyze one. Patients classified into an ordinal outcome based on a synthesis of benefits, harms, QOL Longitudinal snapshot of the patient journey “Exit Examination” or “Discharge Review”

Example RCT with the motivating question: Should we use ceftazidime-avibactam or colistin for the initial treatment of CRE infection?

DOOR DOOR with 4 levels Alive; discharged home Alive; not discharged home; no renal failure Alive; not discharged home; renal failure Death Looking for northward migration of patients in these categories

DOOR DOOR Probability: 64% (55%, 73%) Win Ratio: 3.0 (1.42, 8.39) Colistin (N=63) Caz-Avi (N=30) Discharged home 6 (10%) 6 (20%) Alive; not discharged home; no renal failure 33 (52%) 21 (70%) renal failure 5 (8%) 1 (3%) Death 19 (30%) 2 (7%) DOOR Probability: 64% (55%, 73%) Win Ratio: 3.0 (1.42, 8.39)

DOOR Challenges Cultural change Construction of ordinal outcome is novel and challenging Careful deliberation is essential to synthesize the outcomes An example strategy …

The BAC DOOR ARLG is conducted a pre-trial sub-study to develop DOOR in Staphylococcus aureus bacteremia 20 representative patient profiles (including benefits, harms, and QoL) constructed based on experiences observed in prior trials Profiles sent to 43 expert clinicians. They were asked to rank the patient profiles by desirability of outcome. Examined clinician consensus and component outcomes that drive clinician rankings

SD’s of Patient Rankings

Decision Tree Algorithm Things that we learned Cumulative effect Symptoms important

DOOR Challenges Weighting outcomes / scoring categories was avoided Ranking equates to weighting Concern that composition could hide effects on most important component outcomes Addressing concerns Composite endpoint fundamentals Sensitivity analyses Co-primary endpoints Partial credit

The Future of Clinical Trials Today Tomorrow Endpoints Many Global patient outcome Patient / Clinician Preferences Limited Incorporated Treatment Effects One Many (personalized)

Partial Credit Score Discharged home 100 Alive; not discharged home; no renal failure Partial credit not discharged home; renal failure Death

Partial Credit: How Much? Transparency: pre-specified grading key based on expert survey Patient-guided QoL instrument utilized (higher scores indicate a better QoL) Suppose that in the most desirable category: mean QoL = A Another category: mean QoL score = B Then partial credit for the category = B/A Death = 0

Partial Credit People can disagree. By displaying a treatment contrast as partial credit varies, we can allow people to make their own choices.

Contours of Effects as Partial Credit Varies Category Credit Discharged home 100 Alive; Not discharged home; No renal failure Partial credit Renal failure Death

Binary: Survival Caz-avi advantage: 0.21 (0.06, 0.36) Category Credit Discharged home 100 Alive; Not discharged home; No renal failure Renal failure Death Caz-avi advantage: 0.21 (0.06, 0.36)

Binary: Discharged Home Category Credit Discharged home 100 Alive; Not discharged home; No renal failure Renal failure Death Caz-avi advantage: 0.09 (-0.05, 0.24)

Binary: Alive without Renal Failure Category Credit Discharged home 100 Alive; Not discharged home; No renal failure Renal failure Death Caz-avi advantage: 0.245 (0.06, 0.42)

Compromise Caz-avi advantage: 0.20 (0.07, 0.31) Category Credit Discharged home 100 Alive; Not discharged home; No renal failure 80 Renal failure 60 Death Caz-avi advantage: 0.20 (0.07, 0.31)

The Future of Clinical Trials Today Tomorrow Endpoints Many Global patient outcome Patient / Clinician Preferences Limited Incorporated Treatment Effects One Many (personalized)

Tailoring Medicine Suppose that the new therapy is better. Who benefits from this new therapy…everyone?

Caz-Avi-Colistin Contrast as a Function of Disease Severity DOOR Probability Partial Credit (80/60) Largest differences are in the most severe patients.

We cannot solve problems using the same thinking that we used to create them. Albert Einstein

I hope that you enjoyed this simple exercise. Thank you.