1. Using Outcomes to Analyze Patients Rather than Patients to Analyze Outcomes Scott Evans, MS, PhD Harvard University
SCT
May, 2017

2. Significant Contributors (p<0.001)
Dean Follmann
Thuy Tran
Judith Lok
Michelle Earley
David van Duin

3. Question Suppose a loved one is diagnosed with a serious disease
You are selecting treatment
3 treatment options: A, B, and C
2 outcomes
Treatment success: yes/no
Safety event: yes/no

4. RCT Comparing A, B, and C Analysis of Outcomes
B (N=100)
C (N=100)

5. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
B (N=100)
Success: 50%
C (N=100)
Success: 50%

6. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%

7. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Which treatment would you choose?

8. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Which treatment would you choose?
They all have the same success rate.

9. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Which treatment would you choose?
They all have the same success rate.
A has the lowest safety event rate.

10. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Which treatment would you choose?
They all have the same success rate.
A has the lowest safety event rate.
B and C are indistinguishable.

11. RCT Comparing A, B, and C Analysis of Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Which treatment would you choose?
They all have the same success rate.
A has the lowest safety event rate.
B and C are indistinguishable.
Choose A…right?

12. Analysis of Patients: 4 Possible Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Success
Success
Success
SE + - 15 35 50 50

13. Analysis of Patients: 4 Possible Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Success
Success
Success
SE + - 15 35 50 50

14. Analysis of Patients: 4 Possible Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Success
Success
Success
SE + - 15 35 50 50

15. Analysis of Patients: 4 Possible Outcomes
Success: 50%
Safety event: 30%
B (N=100)
Success: 50%
Safety event: 50%
C (N=100)
Success: 50%
Safety event: 50%
Success
Success
Success
SE + - 15 35 50 50

16. Our culture is to use patients to analyze the outcomes.

17. Our culture is to use patients to analyze the outcomes
Our culture is to use patients to analyze the outcomes. Shouldn’t we use outcomes to analyze the patients?

18. Scott’s father (a math teacher) to his confused son many years ago:
“The order of operations is important…”

19. Instead of getting an optimal solution to the wrong problem, let’s get A solution to the right problem.

20. A Vision

21. The Future of Clinical Trials
Today
Tomorrow
Endpoints
Many
Global patient outcome
Patient / Clinician Preferences
Limited
Incorporated
Treatment Effects
One
Many (personalized)

22. Example
RCT with the motivating question: Should we use ceftazidime-avibactam or colistin for the initial treatment of CRE infection?

23. The Future of Clinical Trials
Today
Tomorrow
Endpoints
Many
Global patient outcome
Patient / Clinician Preferences
Limited
Incorporated
Treatment Effects
One
Many (personalized)

24. Desirability of Outcome Ranking (DOOR)
DOOR with 4 levels
Alive; discharged home
Alive; not discharged home; no renal failure
Alive; not discharged home; renal failure
Death
DOOR probability
The probability of a more desirable global outcome when assigned to the new treatment vs. the standard treatment
Win ratio

25. DOOR DOOR Probability: 64% (55%, 73%) Win Ratio: 3.0 (1.42, 8.39)
Colistin (N=63)
Caz-Avi (N=30)
Discharged home
6 (10%)
6 (20%)
Alive;
not discharged home;
no renal failure
33 (52%)
21 (70%)
renal failure
5 (8%)
1 (3%)
Death
19 (30%)
2 (7%)
DOOR Probability: 64% (55%, 73%)
Win Ratio: 3.0 (1.42, 8.39)

26. The Future of Clinical Trials
Today
Tomorrow
Endpoints
Many
Global patient outcome
Patient / Clinician Preferences
Limited
Incorporated
Treatment Effects
One
Many (personalized)

27. Partial Credit
Score
Discharged home
100
Alive;
not discharged home;
no renal failure
Partial credit
not discharged home; renal failure
Death
For transparency, partial credit can be pre-specified using a survey of experts or obtained from patients during the trial.

28. Contours of Effects as Partial Credit Varies
Category
Credit
Discharged home
100
Alive;
Not discharged home;
No renal failure
Partial credit
Renal failure
Death

29. How much partial credit should be assigned?

30. Binary: Survival Caz-avi advantage: 0.21 (0.06, 0.36) Category Credit
Discharged home
100
Alive;
Not discharged home;
No renal failure
Renal failure
Death
Caz-avi advantage: 0.21 (0.06, 0.36)

31. Binary: Discharged Home
Category
Credit
Discharged home
100
Alive;
Not discharged home;
No renal failure
Renal failure
Death
Caz-avi advantage: 0.09 (-0.05, 0.24)

32. Binary: Alive without Renal Failure
Category
Credit
Discharged home
100
Alive;
Not discharged home;
No renal failure
Renal failure
Death
Caz-avi advantage: (0.06, 0.42)

33. Compromise Caz-avi advantage: 0.20 (0.07, 0.31) Category Credit
Discharged home
100
Alive;
Not discharged home;
No renal failure 80
Renal failure 60
Death
Caz-avi advantage: 0.20 (0.07, 0.31)

34. I am embarrassed by the simplicity of this talk
I am embarrassed by the simplicity of this talk. But I hope that you enjoyed it. Thank you.