1. Another estimation method besides MMRM for average treatment effect in diabetes clinical trials
Yu Du, PhD
Research Scientist
Eli Lilly and Company
Acknowledgment: Bing Liu
Joint Statistical Meeting
Denver, CO
July 30, 2019
Company Confidential ©2019 Eli Lilly and Company

2. Company Confidential ©2019 Eli Lilly and Company
Outlines
Brief intro to MMRM
Brief intro to TMLE
Simulation studies comparing MMRM and TMLE from a hypothetical diabetes phase 3 trial
Next steps and conclusions
Company Confidential ©2019 Eli Lilly and Company

3. Company Confidential ©2019 Eli Lilly and Company
MMRM
MMRM, mixed-model repeated measure, is widely used in estimating treatment effect in diabetes clinical trials as primary analysis model
Mixed-effects model
Response is continuous with repeated measurements, for example, change from baseline in HbA1c
Visits, treatment groups are treated as categorical variables
Treatment group by visit interactions are included
Unstructured within-subject covariance matrix
Company Confidential ©2019 Eli Lilly and Company

4. Company Confidential ©2019 Eli Lilly and Company
TMLE
TMLE, targeted minimum loss based estimation, offers a doubly robust estimation by leveraging the information in longitudinal measurements and baseline variables.
Response can be continuous, binary, count or survival; can be longitudinal
Target only the parameters of interests (e.g., treatment contrast) while treating all others as nuisance parameter (e.g., propensity score)
Aim for smaller bias and variance for the targets at the expense of increased bias and/or variance in the estimation of nuisance parameters.
TMLE creates an estimate that has smaller bias and variance for our target at the expense of increased bias and/or variance in the estimation of nuisance parameters MOA
Company Confidential ©2019 Eli Lilly and Company

5. Company Confidential ©2019 Eli Lilly and Company
MMRM vs TMLE
Both methods have the ability to adjust for the potential bias due to patient dropouts by leveraging the available information.
Which one is better? Bias? Variance? MSE? Coverage probability?
Company Confidential ©2019 Eli Lilly and Company

6. Company Confidential ©2019 Eli Lilly and Company
Simulation Study
The simulation comes from a hypothetical diabetes phase 3 trial (N = 500) where change from baseline in HbA1c is the primary efficacy measure
The “target” is the treatment contrast comparing the treatment to the control group (Randomization ratio 1:1)
Treatment labels are reassigned with Bern(0.5) so that the average treatment contrast is zero
Two dropout scenarios (15% missing)
Missing completely at random
Dropout depends on the treatment arm, baseline value as well as the previous observed HbA1c
10,000 hypothetical trials are simulated for each dropout scenario, and average treatment effect is estimated for each method for each trial.
Keep the unknown relationship between variables
Company Confidential ©2019 Eli Lilly and Company

7. Simulation Results (average treatment effect is 0)
Scenario
Type
Bias
Variance
MSE
RMSE CP
1 MCAR
UNADJ
0.0003 1
0.94
MMRM
0.0002
1.18
TMLE
0.0001
1.29
2 MAR
0.0082
1.12
1.21
In both scenario 1 and 2, TMLE is 9% more efficient than MMRM,
and 29%, 21% more efficient than the unadjusted estimator, respectively.
29% extra efficiency can be translated to approximately 30% saving in sample size!
MCAR : Missing Completely at Random
MAR : Missing at Random
UNADJ : Unadjusted estimator – sample mean difference comparing treatment and control group
RMSE : Relative MSE
CP : Coverage Probability of 95% Confidence Interval
Company Confidential ©2019 Eli Lilly and Company

8. Next steps and conclusions
Compare TMLE and “MMRM” for binary response, for example, whether the patient reaches a certain HbA1c (e.g., 7%) or below at a particular time
Explore the use of machine learning in leveraging an expanded set of baseline variables.
Based on this simulation study, TMLE method is more robust (less bias) to model misspecification and more precise (less variance) than MMRM.
Company Confidential ©2017 Eli Lilly and Company

9. Company Confidential ©2017 Eli Lilly and Company
References
Van Der Laan, M. J., & Rubin, D. (2006). Targeted maximum likelihood learning. The International Journal of Biostatistics, 2(1).
Van Der Laan, M. J., & Gruber, S. (2012). Targeted minimum loss based estimation of causal effects of multiple time point interventions. The international journal of biostatistics, 8(1).
Rosenblum, M., McDermont, A., & Colantuoni, E. (2018). Robust estimation of the average treatment effect in Alzheimer’s disease clinical trials. Biostat bepress.
Company Confidential ©2017 Eli Lilly and Company