1. Predictive Analysis of Clinical Trials Richard Simon, D.Sc. Chief, Biometric Research Branch National Cancer Institute http://brb.nci.nih.gov

2. Biomarker = Biological Measurement Early detection biomarker Endpoint biomarker Prognostic biomarkers Predictive biomarkers

3. Kinds of Biomarkers Endpoint Measured before, during and after treatment to monitor pace of disease and treatment effect Pharmacodynamic (phase 0-1) Does drug hit target Intermediate response (phase 2) Does drug have anti-tumor effect Surrogate for clinical outcome (phase 3)

4. Prognostic biomarkers Measured before treatment to indicate long-term outcome for patients untreated or receiving standard treatment May reflect both disease aggressiveness and effect of standard treatment Used to determine who needs more intensive treatment Predictive biomarkers Measured before treatment to identify who will benefit from a particular treatment

5. Validation = Fitness for Intended Use

6. Single gene or protein measurement Scalar index or classifier that summarizes contributions of multiple genes

7. Prognostic & Predictive Biomarkers in Genomic Oncology Many cancer treatments benefit only a minority of patients to whom they are administered Being able to predict which patients are likely to benefit can Help patients get an effective treatment Help control medical costs Improve the success rate of clinical drug development

8. Biomarker Validity Analytical validity Measures what it’s supposed to Reproducible and robust Clinical validity (correlation) It correlates with something clinically Medical utility Actionable resulting in patient benefit

9. Clinical Utility Prognostic and predictive biomarkers have utility if they are actionable for informing treatment decisions in a manner that results in patient benefit

10. Clinical Utility Biomarker benefits patient by improving treatment decisions Identify patients who have very good prognosis on standard treatment and do not require more intensive regimens Identify patients who are likely or unlikely to benefit from a specific regimen

11. Prognostic markers There is an enormous published literature on prognostic markers in cancer. Very few prognostic markers (factors) are recommended for measurement by ASCO, are approved by FDA or are reimbursed for by payers. Very few play a role in treatment decisions.

12. Pusztai et al. The Oncologist 8:252-8, 2003 939 articles on “prognostic markers” or “prognostic factors” in breast cancer in past 20 years ASCO guidelines only recommend routine testing for ER, PR and HER-2 in breast cancer “With the exception of ER or progesterone receptor expression and HER-2 gene amplification, there are no clinically useful molecular predictors of response to any form of anticancer therapy.”

13. Prognostic Factors in Oncology Most prognostic factors are not used because they are not therapeutically relevant Most prognostic factor studies are not conducted with an intended use clearly in mind They use a convenience sample of patients for whom tissue is available. Generally the patients are too heterogeneous to support therapeutically relevant conclusions There is rarely a validation study separate from the developmental study that addresses medical utility An analytically validated test is rarely developed

14. Prognostic factors for such a heterogeneous group of patients is not “actionable”; i.e. does not help with trteatment decision making.

16. Major problems with prognostic studies of gene expression signatures Inadequate focus on intended use Cases selected based on availability of specimens rather than for relevance to intended use Heterogeneous sample of patients with mixed stages and treatments. Attempt to disentangle effects using regression modeling Too a great a focus on which marker is prognostic or independently prognostic, not whether the marker is effective for intended use

17. If you don’t know where you are going, you might not get there Yogi Berra

18. Prognostic Biomarkers Can be Therapeutically Relevant <10% of node negative ER+ breast cancer patients require or benefit from the cytotoxic chemotherapy that they receive

19. OncotypeDx Recurrence Score Intended use: Patients with node negative estrogen receptor positive breast cancer who are going to receive an anti-estrogen drug following local surgery/radiotherapy Identify patients who have such good prognosis that they are unlikely to derive much benefit from adjuvant chemotherapy

20. Selected patients relevant for the intended use Analyzed the data to see if the recurrence score identified a subset with such good prognosis that the absolute benefit of chemotherapy would at best be very small in absolute terms

21. Biotechnology Has Forced Biostatistics to Focus on Prediction This has led to many exciting methodological developments p>>n problems in which number of genes is much greater than the number of cases And many erroneous publications And growing pains in transitioning from an over- dependence on inference Many of the methods and much of the conventional wisdom of statistics are based on inference problems and are not applicable to prediction problems

22. Goodness of fit is not a proper measure of predictive accuracy Odds ratios and hazards ratios are not proper measures of prediction accuracy Statistical significance of regression coefficients are not proper measures of predictive accuracy

23. Goodness of Fit vs Prediction Accuracy Fit of a model to the same data used to develop it is no evidence of prediction accuracy for independent data “Prediction is difficult; particularly the future.” Dan Quale or Neils Bohr?

25. Prediction on Simulated Null Data Simon et al. J Nat Cancer Inst 95:14, 2003 Generation of Gene Expression Profiles 20 specimens (P i is the expression profile for specimen i) Log-ratio measurements on 6000 genes P i ~ MVN(0, I 6000 ) Can we distinguish between the first 10 specimens (Class 1) and the last 10 (Class 2)? Prediction Method Compound covariate predictor built from the log-ratios of the 10 most differentially expressed genes.

27. Cross Validation Cross-validation simulates the process of separately developing a model on one set of data and predicting for a test set of data not used in developing the model The cross-validated estimate of misclassification error is an estimate of the prediction error for model fit using specified algorithm to full dataset

28. Cross-validation Estimate of Prediction Error 28

29. Cross validation is only valid if the test set is not used in any way in the development of the model. Using the complete set of samples to select genes violates this assumption and invalidates cross-validation. With proper cross-validation, the model must be developed from scratch for each leave-one-out training set. This means that feature selection must be repeated for each leave-one-out training set.

30. Predictive Biomarkers Cancers of a primary site often represent a heterogeneous group of diverse molecular entities which vary fundamentally with regard to the oncogenic mutations that cause them their responsiveness to specific drugs

31. Most cancer treatments benefit only a minority of patients to whom they are administered Being able to predict who requires intensive treatment and who is likely to benefit from which treatments could save patients from unnecessary debilitating adverse effects of treatments that they don’t need or benefit from enhance their chance of receiving a treatment that helps them Help control medical costs Improve the success rate of clinical drug development

32. In most positive phase III clinical trials comparing a new treatment to control, most of the patients treated with the new treatment did not benefit. Adjuvant breast cancer: 70% long-term disease-free survival on control. 80% disease-free survival on new treatment. 70% of patients don’t need the new treatment. Of the remaining 30%, only 1/3 rd benefit.

33. Predictive Biomarkers Estrogen receptor over-expression in breast cancer Anti-estrogens, aromatase inhibitors HER2 amplification in breast cancer Trastuzumab, Lapatinib OncotypeDx gene expression recurrence score in breast cancer Low score for ER+ node - -> no chemotherapy KRAS in colorectal cancer WT KRAS = cetuximab or panitumumab EGFR mutation in NSCLC EGFR inhibitor V600E mutation in BRAF of melanoma vemurafenib ALK translocation in NSCLC crizotinib

34. Standard Paradigm of Broad Eligibility Phase III Clinical Trials Sometimes Leads to Treating many patients with few benefiting Small average treatment effects Problematic for health care economics Inconsistency in results among studies False negative studies

35. The standard approach to designing phase III clinical trials is based on two assumptions Qualitative treatment by subset interactions are unlikely “Costs” of over-treatment are less than “costs” of under-treatment

36. Oncology therapeutics development is now focused on molecularly targeted drugs that are only expected to be effective in a subset of patients whose tumors are driven by the molecular targets Most new cancer drugs are very expensive the aspirin paradigm on which some current clinical trial dogma is based is a roadblock to progress

37. Subset Analysis In the past often studied as un-focused post-hoc analyses Numerous subsets examined Same data used to define subsets for analysis and for comparing treatments within subsets No control of type I error Led to conventional wisdom Only hypothesis generation Only valid if overall treatment difference is significant Only valid if there is a significant treatment by subset interaction

38. Neither current practices of subset analysis nor current practices of ignoring differences in treatment effect among patients are effective for evaluating treatments where qualitative interactions are likely or for informing labeling indications

39. Although the randomized clinical trial remains of fundamental importance for predictive genomic medicine, some of the conventional wisdom of how to design and analyze rct’s requires re-examination The concept of doing an rct of thousands of patients to answer a single question about average treatment effect for a target population presumed homogeneous with regard to the direction of treatment efficacy in many cases no longer has an adequate scientific basis

40. How can we develop new drugs in a manner more consistent with modern tumor biology and obtain reliable information about what regimens work for what kinds of patients?

41. Development is Most Efficient When the Scientific Basis for the Clinical Trial is Strong Having an important molecular target Having an important molecular target Having a drug that can inhibit the target in an overwhelming proportion of tumor cells at an achievable concentration Having a drug that can inhibit the target in an overwhelming proportion of tumor cells at an achievable concentration Having a pre-treatment assay that can identify the patients for whom the molecular target is driving progression of disease Having a pre-treatment assay that can identify the patients for whom the molecular target is driving progression of disease

42. When the Biology is Clear Develop a classifier that identifies the patients likely (or unlikely) to benefit from the new drug Classifier is based on either a single gene/protein or composite score Develop an analytically validated test Measures what it should accurately and reproducibly Design a focused clinical trial to evaluate effectiveness of the new treatment in test + patients

43. Using phase II data, develop predictor of response to new drug Develop Predictor of Response to New Drug Patient Predicted Responsive New Drug Control Patient Predicted Non-Responsive Off Study Targeted (Enrichment) Design

45. Evaluating the Efficiency of Targeted Design Simon R and Maitnourim A. Evaluating the efficiency of targeted designs for randomized clinical trials. Clinical Cancer Research 10:6759-63, 2004; Correction and supplement 12:3229, 2006 Simon R and Maitnourim A. Evaluating the efficiency of targeted designs for randomized clinical trials. Clinical Cancer Research 10:6759-63, 2004; Correction and supplement 12:3229, 2006 Maitnourim A and Simon R. On the efficiency of targeted clinical trials. Statistics in Medicine 24:329-339, 2005. Maitnourim A and Simon R. On the efficiency of targeted clinical trials. Statistics in Medicine 24:329-339, 2005.

46. Relative efficiency of targeted design depends on Relative efficiency of targeted design depends on proportion of patients test positive proportion of patients test positive specificity of treatment effect for test positive patients specificity of treatment effect for test positive patients When less than half of patients are test positive and the drug has minimal benefit for test negative patients, the targeted design requires dramatically fewer randomized patients than the standard design in which the marker is not used When less than half of patients are test positive and the drug has minimal benefit for test negative patients, the targeted design requires dramatically fewer randomized patients than the standard design in which the marker is not used

47. Two Clinical Trial Designs Standard design Randomized comparison of new drug E to control C without the test for screening patients Targeted design Test patients Randomize only test + patients Treatment effect D + in test + patients Treatment effect D - in test – patients Proportion of patients test + is p + Size each design to have power 0.9 and significance level 0.05

48. RandRat = n untargeted /n targeted If D - =0, RandRat = 1/ p + 2 if p + =0.5, RandRat=4 If D - = D + /2, RandRat = 4/(p + +1) 2 if p + =0.5, RandRat=16/9=1.77

49. Comparing T vs C on Survival or DFS 5% 2-sided Significance and 90% Power % Reduction in HazardNumber of Events Required 25%509 30%332 35%227 40%162 45%118 50%88

50. Hazard ratio 0.60 for test + patients Hazard ratio 0.60 for test + patients 40% reduction in hazard 40% reduction in hazard Hazard ratio 1.0 for test – patients Hazard ratio 1.0 for test – patients 0% reduction in hazard 0% reduction in hazard 33% of patients test positive 33% of patients test positive Hazard ratio for unselected population is Hazard ratio for unselected population is 0.33*0.60 + 0.67*1 = 0.87 0.33*0.60 + 0.67*1 = 0.87 13% reduction in hazard 13% reduction in hazard

51. To have 90% power for detecting 40% reduction in hazard within a biomarker positive subset To have 90% power for detecting 40% reduction in hazard within a biomarker positive subset Number of events within subset = 162 Number of events within subset = 162 To have 90% power for detecting 13% reduction in hazard overall To have 90% power for detecting 13% reduction in hazard overall Number of events = 2172 Number of events = 2172

52. Trastuzumab Herceptin Metastatic breast cancer 234 randomized patients per arm 90% power for 13.5% improvement in 1-year survival over 67% baseline at 2-sided.05 level If benefit were limited to the 25% test + patients, overall improvement in survival would have been 3.375% 4025 patients/arm would have been required

53. Web Based Software for Planning Clinical Trials of Treatments with a Candidate Predictive Biomarker http://brb.nci.nih.gov http://brb.nci.nih.gov

55. Principle If a drug is found safe and effective in a defined patient population, approval should not depend on finding the drug ineffective in some other population

56. Implications for Early Phase Studies Need to design and size early phase studies to discover an effective predictive biomarker for identifying the correct target population Need to establish an analytically validated test for measuring the predictive marker in the phase III pivotal studies

57. When the drug is specific for one target and the biology is well understood May need to evaluate several candidate tests e.g. protein expression of target or amplification of gene Need to decide whether to include test negative patients in phase II trials Phase II trials sized for adequate numbers of test positive patients

58. When the drug has several targets or the biology is not well understood Should biologically characterize tumors for all patients on phase II studies with regard to candidate targets and response moderators Phase II trials sized for evaluating candidates Opportunity for sequential and adaptive designs to improve efficiency

59. Empirical screening of expression profiles or mutations to develop predictive marker Larger sample size required Dobbin, Zhao, Simon, Clinical Ca Res 14:108, 2008. Use of archived samples from previous negative phase III trial Use of large disease specific panel of molecularly characterized human tumor cell lines to identify predictive marker

61. “Stratification Design” “Interaction Design” Develop Predictor of Response to New Rx Predicted Non- responsive to New Rx Predicted Responsive To New Rx Control New RXControl New RX

62. Develop prospective analysis plan for evaluation of treatment effect and how it relates to biomarker Defined analysis plan that protects type I error Trial sized for evaluating treatment effect in test + and test – subsets Test negative patients should be adequately protected using interim futility analysis

63. Fallback Analysis Plan Test average treatment effect at reduced level p 0 (e.g..01) If significant claim broad effectiveness If overall effect is not significant, test treatment effect in marker + subset at level.05-p 0 If significant claim effectiveness for marker + subset Test of marker + subset should not require either Overall significance nor Significant interaction 63

64. Sample size for Analysis Plan To have 90% power for detecting uniform 33% reduction in overall hazard at 1% two-sided level requires 370 events. If 33% of patients are positive, then when there are 370 total events there will be approximately 123 events in positive patients 123 events provides 90% power for detecting a 45% reduction in hazard at a 4% two-sided significance level.

68. Bayesian Two-Stage Design RCT With Single Binary Marker 68

69. The Biology is Often Not So Clear Cancer biology is complex and it is not always possible to have the right single predictive classifier identified with an appropriate cut-point by the time the phase 3 trial of a new drug is ready to start accrual Cancer biology is complex and it is not always possible to have the right single predictive classifier identified with an appropriate cut-point by the time the phase 3 trial of a new drug is ready to start accrual

70. The Objectives of a Phase III Clinical Trial Test the strong null hypothesis that the test treatment is uniformly ineffective compared to control for primary endpoint If the null hypothesis is rejected, develop a labeling indication for informing physicians in their decisions about which patients they treat with the drug.

71. The test of the null hypothesis of no average treatment effect is not necessarily a good test of the strong null hypothesis that the new treatment is uniformly ineffective Rejection of the null hypothesis is not in itself adequate information for guiding physicians on how to use the treatment

72. 72 Biomarker Selection Design Based on Adaptive Threshold Design W Jiang, B Freidlin & R Simon JNCI 99:1036-43, 2007

73. Biomarker Selection Design Have identified K candidate biomarkers B 1, …, B K thought to be predictive of patients likely to benefit from T relative to C Have identified K candidate biomarkers B 1, …, B K thought to be predictive of patients likely to benefit from T relative to C Cut-points not necessarily established for each biomarker Cut-points not necessarily established for each biomarker Eligibility not restricted by candidate markers Eligibility not restricted by candidate markers

74. Marker Selection Design

75. Compute p* = min{p 1, p 2, …, p K } Compute whether the value of p* is statistically significant when adjusted for multiple testing Adjust for multiple testing by permuting the treatment labels and re-calculating p 1 …p K and p* for the permuted treatment labels Repeat for 10,000 random permutations to approximate the null distribution of p*

76. To detect a 40% reduction in hazard in an a-priori defined subset with 90% power and a 4% two-sided significance level requires 171 events in the subset. To adjust for multiplicity with 4 independent binary tests, 171 -> 224. If 33% are positive for each marker, then the trial might be sized for 3*224= total 672 events.

77. Designs When there are Many Candidate Markers and too Much Patient Heterogeneity for any Single Marker

79. Adaptive Signature Design

80. The indication classifier is not a binary classifier of whether a patient has good prognosis or poor prognosis It is a “two sample classifier” of whether the prognosis of a patient on E is better than the prognosis of the patient on C

81. The indication classifier can be a binary classifier that maps the vector of candidate covariates into {E,C} indicating which treatment is predicted superior for that patient The classifier need not use all the covariates but variable selection must be determined using only the training set Variable selection may be based on selecting variables with apparent interactions with treatment, with cut-off for variable selection determined by cross-validation within training set for optimal classification The indication classifier can be a probabilistic classifier 81

82. 82

83. 83

84. Treatment effect restricted to subset. 10% of patients sensitive, 400 patients. TestPower Overall.05 level test46.7 Overall.04 level test43.1 Sensitive subset.01 level test (performed only when overall.04 level test is negative) 42.2 Overall adaptive signature design85.3

85. Overall treatment effect, no subset effect. 400 patients TestPower Overall.05 level test74.2 Overall.04 level test70.9 Sensitive subset.01 level test1.0 Overall adaptive signature design70.9

86. This approach can be used with any set of candidate predictor variables This approach can also be used to identify the subset of patients who don’t benefit from E in cases where E is superior to C overall This approach can also be used to identify the subset of patients who don’t benefit from E in cases where E is superior to C overall

89. Cross-Validated Adaptive Signature Design Define indication classifier development algorithm A Apply algorithm to full dataset to develop indication classifier for use in future patients M(x;A,P) Using K fold cross validation Classify patients in test sets based on classifiers developed in training sets; e.g. y i =M(x i ;A,P -i ) S={i : y i = E} Compare E to C in S and estimate size of treatment effect is an estimate of the size of the treatment effect for future patients with M(x;A,P)=E

90. Cross-Validated Adaptive Signature Design Approximate null distribution of Permute treatment labels Repeat complete cross-validation procedure Generate permutation distribution of the values for permuted data Test null hypothesis that the treatment effect in classifier positive patients is null using as test statistic cross-validated estimate of treatment effect in positive patients

91. Key Ideas Replace multiple significance testing by development of one indication classifier Control study-wise type I error for significance test of Overall average treatment effect Treatment effect in classifier positive patients Test of treatment effect in classifier positive patients does not depend on significance of overall test nor on significant interaction Obtain unbiased or conservative estimate of the treatment effect of future classifier positive patients

92. The size of the E vs C treatment effect for the indicated population is (conservatively) estimated from the cross validation by the Kaplan Meier survival curves of E and of C in S The Kaplan-Meier curves of E and C for patients in S provides an estimate of

93. The stability of the indication classifier M(x;A,D)can be evaluated by examining the consistency of classifications M(x i ;A, B) for bootstrap samples B from D.

94. Although there may be less certainty about exactly which types of patient benefit from E relative to C, classification may be better than for many standard clinical trial in which all patients are classified based on results of testing the single overall null hypothesis

95. 70% Response to E in Sensitive Patients 25% Response to E Otherwise 25% Response to C 30% Patients Sensitive ASDCV-ASD Overall 0.05 Test 0.8300.838 Overall 0.04 Test 0.7940.808 Sensitive Subset 0.01 Test 0.3060.723 Overall Power 0.8250.918

96. 25% Response to T 25% Response to C No Subset Effect ASDCV-ASD Overall 0.05 Test 0.0470.056 Overall 0.04 Test 0.040.048 Sensitive Subset 0.01 Test 0.0010 Overall Power 0.0410.048

97. For Binary Outcome

100. Covariates: Age, performance status (pf), tumor size (sz), stage/grade index (sg), serum acid phosphatase (ap) 506 prostate cancer patients were randomly allocated to one of four arms: Placebo and 0.2 mg of diethylstilbestrol (DES) were combined as control arm C 1.0 mg DES, or 5.0 mg DES were combined as T. The end-point was overall survival (death from any cause).

101. Figure 1: Overall analysis. The value of the log-rank statistic is 2.9 and the corresponding p-value is 0.09. The new treatment thus shows no benefit overall at the 0.05 level.

102. Figure 2: Cross-validated survival curves for patients predicted to benefit from the new treatment. log-rank statistic = 10.0, permutation p-value is.002

103. Figure 3: Survival curves for cases predicted not to benefit from the new treatment. The value of the log-rank statistic is 0.54.

107. Prediction Based Clinical Trials We can evaluate our methods for analysis of clinical trials in terms of their effect on patient outcome via informing therapeutic decision making

108. Expected Survival Distribution for Future Patients With Standard Analysis

109. Expected Survival Distribution for Future Patients With Indication Classifier

110. Hence, alternative methods for analyzing RCT’s can be evaluated in an unbiased manner with regard to their value to patients using the actual RCT data

111. Conclusions New biotechnology and knowledge of tumor biology provide important opportunities to improve therapeutic decision making Treatment of broad populations with regimens that do not benefit most patients is increasingly no longer necessary nor economically sustainable The established molecular heterogeneity of human diseases requires the use new approaches to the development and evaluation of therapeutics

112. Acknowledgements Boris Freidlin Wenyu Jiang Aboubakar Maitournam Jyothi Subramanian