1. A Vision for Clinical Trials Scott Evans, MS, PhD Harvard University
May, 2017

2. Significant Contributors (p<0.001)
Dean Follmann
Dan Rubin
Chip Chambers
Thuy Tran
Judith Lok
Michelle Earley
David van Duin

3. Most clinical trials fail to provide the evidence needed to inform medical decision-making. However, the serious implications of this deficit are largely absent from public discourse.
DeMets and Califf, JAMA, 2011

4. The medical community is calling for more:
Systematic evaluation of benefits and harms
Pragmatism

5. If I had one hour to solve a problem, I would spend 55 minutes defining and understanding the problem, and 5 minutes solving it.
Albert Einstein

6. Question 1 Suppose we measure the duration of hospitalization
Shorter duration is better … or is it?
The faster the patient dies, the shorter the duration
Interpretation of an outcome need clinical context of other outcomes for the same patient

7. Question 2 We define analysis populations
Efficacy analysis: efficacy population (e.g., ITT)
Safety analysis: safety population
Efficacy population ≠ safety population
We combine these analyses into benefit:risk analyses
To whom does this analysis apply?

8. Question 3 Suppose a loved one is diagnosed with a serious disease
You are selecting treatment
3 treatment options: A, B, and C
2 outcomes
Treatment success: yes/no
Safety event: yes/no

9. RCT Comparing A, B, and C Analysis of Outcomes
B (N=100)
C (N=100)

10. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
B (N=100)
Success: 50%
C (N=100)
Success: 50%

11. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%

12. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Which treatment would you choose?

13. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Which treatment would you choose?
They all have the same success rate.

14. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Which treatment would you choose?
They all have the same success rate.
A has the lowest safety event rate.

15. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Which treatment would you choose?
They all have the same success rate.
A has the lowest safety event rate.
B and C are indistinguishable.

16. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Which treatment would you choose?
They all have the same success rate.
A has the lowest safety event rate.
B and C are indistinguishable.
Choose A…right?

17. Analysis of Patients: 4 Possible Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Success
Success
Success
SE + - 15 35 50 50

18. Analysis of Patients: 4 Possible Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Success
Success
Success
SE + - 15 35 50 50

19. Analysis of Patients: 4 Possible Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Success
Success
Success
SE + - 15 35 50 50

20. Analysis of Patients: 4 Possible Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Success
Success
Success
SE + - 15 35 50 50

21. Our culture is to use patients to analyze the outcomes.

22. Our culture is to use patients to analyze the outcomes
Our culture is to use patients to analyze the outcomes. Shouldn’t we use outcomes to analyze the patients?

23. Scott’s father (a math teacher) to his confused son many years ago:
“The order of operations is important…”

24. Instead of getting an optimal solution to the wrong problem, let’s get A solution to the right problem.

25. Translational Statistics
Conventional analysis provides fragmented information
Lack of synthesis of the totality of evidence / total disease burden
Difficult to use trial results to inform patient management
Let’s design and analyze studies to have greater applicability for clinical practice

26. A Vision

27. The Future of Clinical Trials
Today
Tomorrow
Endpoints
Many
Global patient outcome
Patient / Clinician Preferences
Limited
Incorporated
Treatment Effects
One
Many (personalized)

28. The Future of Clinical Trials
Today
Tomorrow
Endpoints
Many
Global patient outcome
Patient / Clinician Preferences
Limited
Incorporated
Treatment Effects
One
Many (personalized)

30. Pairwise Comparison of Patients
Rank patients based on a global patient outcomes synthesizing benefits and harms
Estimate
Win ratio: # wins / # losses
DOOR probability: the probability of a more desirable global outcome when assigned to the new vs. the standard treatment

31. Desirability of Outcome Ranking (DOOR)
Before we analyze several hundred patients,
we must understand how to analyze one.
Patients classified into an ordinal outcome based on a synthesis of benefits, harms, QOL
Longitudinal snapshot of the patient journey
“Exit Examination” or “Discharge Review”

32. Example
RCT with the motivating question: Should we use ceftazidime-avibactam or colistin for the initial treatment of CRE infection?

33. DOOR DOOR with 4 levels Alive; discharged home
Alive; not discharged home; no renal failure
Alive; not discharged home; renal failure
Death
Looking for northward migration of patients in these categories

34. DOOR DOOR Probability: 64% (55%, 73%) Win Ratio: 3.0 (1.42, 8.39)
Colistin (N=63)
Caz-Avi (N=30)
Discharged home
6 (10%)
6 (20%)
Alive;
not discharged home;
no renal failure
33 (52%)
21 (70%)
renal failure
5 (8%)
1 (3%)
Death
19 (30%)
2 (7%)
DOOR Probability: 64% (55%, 73%)
Win Ratio: 3.0 (1.42, 8.39)

35. DOOR Challenges Cultural change
Construction of ordinal outcome is novel and challenging
Careful deliberation is essential to synthesize the outcomes
An example strategy …

36. The BAC DOOR
ARLG is conducted a pre-trial sub-study to develop DOOR in Staphylococcus aureus bacteremia
20 representative patient profiles (including benefits, harms, and QoL) constructed based on experiences observed in prior trials
Profiles sent to 43 expert clinicians. They were asked to rank the patient profiles by desirability of outcome.
Examined clinician consensus and component outcomes that drive clinician rankings

37. SD’s of Patient Rankings

38. Decision Tree Algorithm
Things that we learned
Cumulative effect
Symptoms important

39. DOOR Challenges Weighting outcomes / scoring categories was avoided
Ranking equates to weighting
Concern that composition could hide effects on most important component outcomes
Addressing concerns
Composite endpoint fundamentals
Sensitivity analyses
Co-primary endpoints
Partial credit

40. The Future of Clinical Trials
Today
Tomorrow
Endpoints
Many
Global patient outcome
Patient / Clinician Preferences
Limited
Incorporated
Treatment Effects
One
Many (personalized)

41. Partial Credit Score Discharged home 100 Alive; not discharged home;
no renal failure
Partial credit
not discharged home; renal failure
Death

42. Partial Credit: How Much?
Transparency: pre-specified grading key based on expert survey
Patient-guided
QoL instrument utilized (higher scores indicate a better QoL)
Suppose that in the most desirable category: mean QoL = A
Another category: mean QoL score = B
Then partial credit for the category = B/A
Death = 0

43. Partial Credit
People can disagree. By displaying a treatment contrast as partial credit varies, we can allow people to make their own choices.

44. Contours of Effects as Partial Credit Varies
Category
Credit
Discharged home
100
Alive;
Not discharged home;
No renal failure
Partial credit
Renal failure
Death

45. Binary: Survival Caz-avi advantage: 0.21 (0.06, 0.36) Category Credit
Discharged home
100
Alive;
Not discharged home;
No renal failure
Renal failure
Death
Caz-avi advantage: 0.21 (0.06, 0.36)

46. Binary: Discharged Home
Category
Credit
Discharged home
100
Alive;
Not discharged home;
No renal failure
Renal failure
Death
Caz-avi advantage: 0.09 (-0.05, 0.24)

47. Binary: Alive without Renal Failure
Category
Credit
Discharged home
100
Alive;
Not discharged home;
No renal failure
Renal failure
Death
Caz-avi advantage: (0.06, 0.42)

48. Compromise Caz-avi advantage: 0.20 (0.07, 0.31) Category Credit
Discharged home
100
Alive;
Not discharged home;
No renal failure 80
Renal failure 60
Death
Caz-avi advantage: 0.20 (0.07, 0.31)

49. The Future of Clinical Trials
Today
Tomorrow
Endpoints
Many
Global patient outcome
Patient / Clinician Preferences
Limited
Incorporated
Treatment Effects
One
Many (personalized)

50. Tailoring Medicine
Suppose that the new therapy is better. Who benefits from this new therapy…everyone?

51. Caz-Avi-Colistin Contrast as a Function of Disease Severity
DOOR Probability
Partial Credit (80/60)
Largest differences are in the most severe patients.

52. We cannot solve problems using the same thinking that we used to create them.
Albert Einstein

53. I hope that you enjoyed this simple exercise. Thank you.