Evaluating the “One-Model Fits All” Approach for Modeling Clinical Trial Adverse events Stephanie Pan, MS Icahn School of Medicine at Mount Sinai Hospital, New York, NY

Background In an RCT, AEs are collected to monitor patient safety. These data are typically collapsed into counts per patient and summarized as rates over patient time at risk. Although the distribution of events per patient will vary widely depending on AE type, treatment differences are typically estimated assuming the same underlying distributional assumption for all AEs (typically Poisson). However, what happens if the distributional assumptions are not met and how robust are these count models to misspecifications?

Simulation Method Negative Binomial (NB) Zero-Inflated Negative Binomial (ZINB) Poisson Inverse Gaussian (PIG) Poisson Hurdle (PH) Zero-Inflated Poisson (ZIP) Negative Binomial Hurdle (NBH) We evaluated the robustness of the typical Poisson model versus extensions and alternatives under various assumptions using simulation.  Alternatives explored include: Generated 1000 repeated samples of AE counts from 500 patients assuming AEs arise from true Poisson, over-dispersed NB, zero-inflated, and zero-count with positive-count process distributions Assumed 20% structural zeroes in simulating zero-inflated and zero-count process Fit various models and compared on mean estimates, mean squared error (MSE), and coverage probability (how often does the 95% confidence interval include the true estimate) We then fit these models to AE data from a recent RCT to assess performance and fit. 

True Distribution (β=1.60) Simulation Results Mean estimate and coverage probabilities are reported in the table below: True Distribution (β=1.60) Poisson NB* NB** ZIP ZINB* 1.61 (0.95) 1.60 (0.82) 1.61 (0.55) 1.62 (0.94) 1.60 (0.83) NB 1.60 (0.96) 1.61 (0.94) 1.62 (0.95) 1.60 (0.95) PIG --- 1.57 (0.96) 1.33 (0.92) 1.55 (0.96) 1.49 (0.96) 1.32 (0.73) 1.13 (0.42) 1.56 (0.88) 1.35 (0.75) ZINB 1.57 (0.94) 1.58 (0.92) 1.55 (0.90) 1.60 (0.91) Poisson Hurdle 2.09 (0.97) 2.35 (0.97) 1.36 (0.75) NB Hurdle 2.11 (0.97) 1.64 (0.82) 1.57 (0.93) 2.38 (0.97) 1.67 (0.94) Poisson mean estimate is consistent across the varying distributions However, coverage probability for Poisson model in the presence of overdispersion is reduced Poisson model had consistently produced estimated standard errors that were too small PIG model is comparable to NB but failed to converge for simulated Poisson process distributions *Low dispersion (k=0.25) **High dispersion (k=0.05)

Zero-Count, Zero Truncated Poisson Zero-Count, Zero Truncated NB* Simulation Results Mean estimate and coverage probabilities are reported in the table below: Important to consider if zeroes are from a single “structural” source and if more flexible models are needed to handle zero counts If the true underlying distribution follows a zero-count, zero truncated Poisson we find that the Poisson and NB hurdle models are comparable in estimation and coverage If the true underlying distribution follows a zero-count, zero truncated NB we find that the Poisson hurdle model performs poorly in estimation True Distribution (β=1.60) Zero-Count, Zero Truncated Poisson Zero-Count, Zero Truncated NB* Poisson 0.30 (0) 0.66 (0) NB PIG --- 0.61 (0) ZIP 0.79 (0) ZINB 0.68 (0) Poisson Hurdle 1.63 (0.95) 1.27 (0.27) NB Hurdle 1.64 (0.96) 1.60 (0.94) Important to consider if zeroes are from a single “structural” source and if more flexible models are needed to handle zero counts *Low dispersion (k=0.25)

Application To AN RCT Cardiothoracic Surgical Network (CTSN) Trial on Intramyocardial Injection of Mesenchymal Precursor Cells (MPCs) among Left Ventricular Assist Device (LVAD) recipients with advanced heart failure Patients (N=159) were randomized to receive injections of MPCs or placebo and adverse events were reported at 6 months Population of patients were more likely to experience certain type of AEs such as bleeding, infections, cardiac arrhythmias Model estimates, fit, and performance were assessed using 1000 repeated sub-sampling and cross-validation Poisson AIC was consistently high in the presence of specific AEs with overdispersion In some instances, robust or sandwich estimators can be used to adjust SEs Poisson Distribution of MSE for Overall Bleeding Poisson Distribution of AIC for Overall Bleeding

Conclusion and Ongoing Research Explored extended count models such as PIG, zero-Inflated, and hurdle with applications to an RCT Prior studies have shown that the Poisson models, including ZIP and Poisson hurdle, have poorer coverage probability in the presence of underlying overdispersion Given the potential cost of misspecification, careful consideration is needed for a “One-Model Fits All” approach and the functional form of the count responses Further research is needed on incorporating a composite effect size across multiple AE models versus modeling the total number of AEs Acknowledgements Jessica Overbey, MS and Emilia Bagiella, PhD for their thoughtful insights and helpful review of the abstract and presentation.

References Hilbe, J. (2014). Modeling Count Data. Cambridge: Cambridge University Press. doi:10.1017/CBO9781139236065 Yau TM, Pagani FD, Mancini DM, et al. Intramyocardial Injection of Mesenchymal Precursor Cells and Successful Temporary Weaning From Left Ventricular Assist Device Support in Patients With Advanced Heart Failure: A Randomized Clinical Trial. JAMA. 2019;321(12):1176–1186. doi:10.1001/jama.2019.2341 Horton NJ, Kim E, Saitz R. A cautionary note regarding count models of alcohol consumption in randomized controlled trials. BMC Med Res Methodol. 2007;7:9. Published 2007 Feb 15. doi:10.1186/1471-2288-7-9 Hu MC, Pavlicova M, Nunes EV. Zero-inflated and hurdle models of count data with extra zeros: examples from an HIV-risk reduction intervention trial. Am J Drug Alcohol Abuse. 2011;37(5):367–375. doi:10.3109/00952990.2011.597280 Cameron AC, Trivedi PK. Regression analysis of count data. Cambridge, UK: Cambridge University Press; 1998.