1. Xin Huang, Yan Sun, V. Devanarayan Exploratory Statistics AbbVie Inc.
Paths to precision medicine: Subgroup Identification in Clinical Trials
Xin Huang, Yan Sun, V. Devanarayan
Exploratory Statistics
AbbVie Inc.
April. 15, 2015

2. Why Personalized Medicine?
Edward Abrahams and Mike Silver. The Case for Personalized Medicine. (2009) Journal of Diabetes Science and Technology V3 Issue 4

3. Some statistical challenges
For ease of implementation in clinical practice, need cut-points on biomarkers for predicting responders/non-responders.
i.e., threshold-based biomarker signatures
E.g., Patients with Gene X1 > …, Gene X2 < …, are likely responders.
This should be “Multivariate”.
Derive this from typically a large panel of candidate markers, and often from full genome data (e.g., > 30,000 genes).
Need to account for linear and nonlinear trends.
After a promising threshold-based signature is identified, need to predict it’s performance in a future dataset.
i.e., predict treatment effect in the “responder” subgroup, or predict the signature effect among patients receiving treatment.

4. Biomarker signatures for subgroup identification
Prognostic Signature:
identifies a subgroup of patients that are more likely to experience an outcome of interest (efficacy, toxicity, disease progression, etc.), independent of treatment.
Predictive Signature:
identifies a subgroup of patients that respond better to a specific treatment.

5. Some existing methods
Prognostic Signatures (predict the disease outcome irrespective of the treatment ):
CART (Breiman et al, 1984)
MARS (Friedman, 1991)
RuleFit (Friedman and Popescu 2008)
Predictive Signatures (predict the response to a specific treatment compared to other treatments):
Interaction Trees (Su et al. 2008, 2009)
Virtual Twins ( Foster et al. 2011)
SIDES method (Lipkovich et al. 2011, 2014)
Bayesian approaches (Berger et al. 2014)

6. Objective functions
Consider a supervised learning problem with data 𝒙 𝒊 , 𝑦 𝑖 , 𝑖= 1, 2, …, 𝑛, where 𝒙 𝒊 is a p-vector of predictor and 𝑦 𝑖 is an outcome variable
Consider three major applications:
Linear regression for continuous response
Logistic regression for binary response, where 𝑦 𝑖 ∈ 0, 1
Cox regression for survival response: 𝑦 𝑖 =( 𝑇 𝑖 , 𝛿 𝑖 ), where 𝑇 𝑖 is a right censored survival time and 𝛿 𝑖 is the censoring indicator
Denote the log likelihood or log partial likelihood by 𝓁(𝜂;𝑿, 𝒚), where 𝜂 is the usual linear combination of predictors.
continuous response in simple linear regression
log odds in logistic regression
log hazard ratio in proportional hazards regression.

7. Objective functions, contd.
Consider the following model for prognostic signatures (predict the outcome, irrespective of the treatment),
𝜂=𝛼 + 𝛽∙𝜔(𝑿), (1)
where 𝜔 𝑿 ={0, 1} is the signature rule returning grouping indicators for each subject.
Consider following model for predictive signatures (predict the response to a specific treatment compared to the other treatment),
𝜂=𝛼 + 𝛽∙ 𝜔 𝑿 ×𝑟 + 𝛾∙𝑟, (2)
where r is the treatment indicator.
Our algorithms derive signature rules, 𝜔 𝑿 , with the objective of searching for a best grouping to optimize the significance of 𝛽 in (1) and (2)

8. Bootstrapping & Aggregating of Thresholds from Trees (BATTing)
Original Data
Bootstrapping (sampling with replacement)
Data 1
Data 2
Data B
… … ...
Tree 1
>= C1
< C1
Tree 2
>= C2
< C2
… … ...
Tree B
>= CB
< CB
Aggregate Thresholds (C1, C2, …., CB)
Threshold is robust to small perturbations in data, outliers, etc.
BATTing Threshold (Median)
(Devanarayan, 1999)

9. BATTing, contd.
We use the same simulation model as described in the simulation section for prognostic signatures except that we only include one true predictor as the only candidate predictor. Under the simulated model, the overall response equals to zero, and the signature positive group (cutoff >= 0) has a positive response, while the signature negative group (cutoff < 0) has a negative response. The difference between the two signature groups is determined by a predetermined effect size.
Figure 1 shows the distribution of BATTing threshold estimates from 500 simulation runs across different number of bootstrapping for sample size = 100 and effect size = 0.2, with true cutoff being 0 (red dashed vertical line). As demonstrated in Figure 1, BATTing helps reduce the influence of data perturbations in the dataset and thus stabilize the threshold estimate. In our experience, the number of bootstraps >= 50 is recommended.

10. Sequential BATTing Model Growing within the potential Sig+ group
Marker 7
Marker 3
Marker 9
Whole Population
(Sig+)
(Sig+)
(Sig+)
Sig+
Sig-
Sig-
Sig-
Sig-
Model Growing within the potential Sig+ group
Get the BATTing threshold for each unused marker
The best marker is selected to split the current sig+ group
This procedure continues in the new Sig+ group
Stopping Rule:
The new added predictor goes through the likelihood ratio test for significance.

11. Adaptive Index Model
AIM (Tian & Tibshirani, 2010) can be used for selecting markers & thresholds.
Output: AIM Score
An index predictor: # of satisfied rules
𝒔𝒄𝒐𝒓𝒆= 𝒌=𝟏 𝑲 𝑰( 𝑋 𝑘 ≤ 𝑐 𝑘 )
Model to get the AIM score
Prognostic: 𝜂 ∗ = 𝜃 0 +𝜽×𝒔𝒄𝒐𝒓𝒆,
Predictive: 𝜂 ∗ = 𝜃 0 +𝛾∙𝑇+ 𝜽×𝑻×𝒔𝒄𝒐𝒓𝒆.
An information matrix based fast algorithm is used to do score test to select threshold for each marker
Markers are selected one at a time (forward selection)
Optimal # of markers is determined via cross validation

12. AIM-BATTing Obtain the AIM Score
Use BATTing to derive an optimal AIM Score threshold based on Model (1) & (2). The threshold is then used to stratify the population.
Step1
Step2
Patient 1
AIM
I(X1≥c1) +
I(X2≤c2)
…..
I(Xk≥ck)
Score 1
Sig+ Grp.
Patient 2
Score 2
BATTing
I( Score ≥ j )
Sig- Grp.
Patient n
Score n

13. Some Refinements to the AIM-BATTing algorithm
MC-AIM-BATTing:
Monte Carlo procedure to get a more stable estimate of the “optimal # of markers”.
i.e., use the median of estimated “optimal # of markers” across multiple cross validation runs with different random seeds
MC-AIM-RULE-BATTing:
Use BATTing directly on the rules (Xi > c), instead of scores, and get a cutoff on the rule list.
Patients meeting all the rules within the cutoff are assigned to the sig+ group

14. Performance evaluation: Common mistakes in practice
Using an entire dataset to build a model Select “important” variables by associating markers with outcomes (e.g., stepwise regression) Test and rely on lack of fit assessment of the resulting model Assuming the resulting model is correct, making inferences using the same dataset
over-fitting

15. Predictive significance via cross-validation
Train
Train
Train
Sig.
Sig.
Sig.
Test
Test
Test
Repeat Multiple Times
Group Label
Group Label
Group Label
Group Label
Group Label
CV p-value pi
Aggregated cross-validated p values from M iterations (p1, p2, …., pM)
predictive significance (median of p value)
Note: other performance statistics, e.g., sensitivity, specificity, PPV, NPV, hazard ratio, odds ratio can be calculated similarly

16. Effect size = E(Y|Trt, sig+) - E(Y|ctrl, sig+) = 0.5
Simulation Design
Similar simulation model as Lipkovich et al., 2011, 2014, with each predictor as continuous instead of dichotomized valued
Small trials to large trials (n=100, 300, 500)
Number of candidate predictors is k=10 and 18 with different correlation structures
Effect size is 0.2 (low), 0.5 (medium), 0.8 (high)
Effect size = E(Y|Trt, sig+) - E(Y|ctrl, sig+) = 0.5
0.5

17. Simulation Results
For small effect size, none of the methods has many testing p values less than 0.05 for sample size from 100 to 500
Our proposed methods outperform SIDES in terms of the selection accuracy: the accuracy of SIDES is around 50% while that of our proposed algorithms is from 60% to 70% for large sample size.
For effect size greater than medium (0.5) and sample size larger than 300, our proposed methods have most of the testing p values less than 0.05 and accuracy around 90%. SIDES method under performs in all scenarios.

18. Clinical Trial Case Study
Data simulated based on a Phase III clinical trial
Efficacy of a novel treatment is compared to the standard of care (Control) in patients with severe sepsis
Treatment arm (n = 317) vs. Control arm (n = 153)
Outcome: Binary (survival)
Available markers:
demographic and clinical covariates, i.e., age, time from first sepsis-organ fail to start drug, sum of baseline SOFA socres (cardiovascular, hematology, hepaticrenal, and respiration scores), number of baseline organ failures, pre-infusion apache-ii score, baseline GLASGOW coma scale score, baseline activity of daily living score
laboratory markers, i.e., baseline local platelets, creatinine, serum IL-6 concentration, local bilirunbin
Overall outcome was insignificant (1-tailed p value = 0.08), with survival rates of 40.7% and 34% in the treatment and control arms, respectively
Original data was randomly split into two parts (training + testing)

19. Clinical Trial Case Study, contd.
Signature rules for positive subgroup:
Sequential-BATTing and AIM-RULE: pre-infusion apache-ii score <= 27
AIM: meet at least two out of the three thresholds: (1) pre-infusion apache-ii score < 27; (2) Age < 54; (3) local bilirunbin > 0.8
SIDES: Creatinine <= 1.1 & baseline GLASGOW coma scale score > 11
Table: 1-tailed p-values for sepsis trial example
Seq-BATTing has the most promising CV performance, and its signature is validated in the test dataset

20. Summary
The proposed subgroup identification algorithms perform well in simulations and case-study illustration.
These algorithms provide threshold-based multivariate biomarker signatures.
Variable selection is automatically built-in to these algorithms.
Personalized medicine is a paradigm shift in drug development, which requires
Advanced subgroup identification and subgroup analysis methods
Enrichment design and simulations
Smart diagnostic test development and clinical development strategy to overcome operational challenges
Collaboration between functional areas

21. References
Hastie T, Tibshirani R, Friedman J (2011) The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition, 2nd ed Corr. 7th printing 2013 edition. Springer
Breiman L, Friedman J, Stone CJ, Olshen RA (1984) Classification and Regression Trees, 1 edition. Chapman and Hall/CRC
Friedman JH (1991) Multivariate Adaptive Regression Splines. Ann Stat 19:1–67. doi: /aos/
Friedman JH, Popescu BE (2008) Predictive learning via rule ensembles. Ann Appl Stat 2:916–954. doi: /07-AOAS148
Liu X, Minin V, Huang Y, et al. (2004) Statistical methods for analyzing tissue microarray data. J Biopharm Stat 14:671–685. doi: /BIP
Chen G, Zhong H, Belousov A, Devanarayan V (2015) A PRIM approach to predictive-signature development for patient stratification. Stat Med 34:317–342. doi: /sim.6343
Su X, Zhou T, Yan X, et al. (2008) Interaction Trees with Censored Survival Data. Int J Biostat. doi: /
Su X, Tsai C-L, Wang H, et al. (2009) Subgroup Analysis via Recursive Partitioning. J Mach Learn Res 10:141–158.
Lipkovich I, Dmitrienko A, Denne J, Enas G (2011) Subgroup identification based on differential effect search--a recursive partitioning method for establishing response to treatment in patient subpopulations. Stat Med 30:2601–2621. doi: /sim.4289
Lipkovich I, Dmitrienko A (2014) Strategies for identifying predictive biomarkers and subgroups with enhanced treatment effect in clinical trials using SIDES. J Biopharm Stat 24:130–153. doi: /
Berger JO, Wang X, Shen L (2014) A Bayesian approach to subgroup identification. J Biopharm Stat 24:110–129. doi: /
Devanarayan V, Cummins D, Tanzer L (1999) Application of GAM and tree models for assessing the role of drug resistance proteins in leukemia chemotherapy.
Tian L, Tibshirani R (2011) Adaptive index models for marker-based risk stratification. Biostatistics 12:68–86. doi: /biostatistics/kxq047
Tian L, Alizadeh A, Gentles A, Tibshirani R (2012) A Simple Method for Detecting Interactions between a Treatment and a Large Number of Covariates. arXiv
Tibshirani R, Efron B (2002) Pre-validation and inference in microarrays. Stat Appl Genet Mol Biol. doi: /
Foster JC, Taylor JM, Ruberg SJ (2011) Subgroup identification from randomized clinical trial data. Stat Med. 30(24)