1. Evaluating the “One-Model Fits All” Approach for Modeling Clinical Trial Adverse events
Stephanie Pan, MS
Icahn School of Medicine at Mount Sinai Hospital, New York, NY

2. Background
In an RCT, AEs are collected to monitor patient safety. These data are typically collapsed into counts per patient and summarized as rates over patient time at risk. Although the distribution of events per patient will vary widely depending on AE type, treatment differences are typically estimated assuming the same underlying distributional assumption for all AEs (typically Poisson). However, what happens if the distributional assumptions are not met and how robust are these count models to misspecifications?

3. Simulation Method Negative Binomial (NB)
Zero-Inflated Negative Binomial (ZINB)
Poisson Inverse Gaussian (PIG)
Poisson Hurdle (PH)
Zero-Inflated Poisson (ZIP)
Negative Binomial Hurdle (NBH)
We evaluated the robustness of the typical Poisson model versus extensions and alternatives under various assumptions using simulation.  Alternatives explored include:
Generated 1000 repeated samples of AE counts from 500 patients assuming AEs arise from true Poisson, over-dispersed NB, zero-inflated, and zero-count with positive-count process distributions
Assumed 20% structural zeroes in simulating zero-inflated and zero-count process
Fit various models and compared on mean estimates, mean squared error (MSE), and coverage probability (how often does the 95% confidence interval include the true estimate)
We then fit these models to AE data from a recent RCT to assess performance and fit. 

4. True Distribution (β=1.60)
Simulation Results
Mean estimate and coverage probabilities are reported in the table below:
True Distribution (β=1.60)
Poisson
NB*
NB**
ZIP
ZINB*
1.61 (0.95)
1.60 (0.82)
1.61 (0.55)
1.62 (0.94)
1.60 (0.83) NB
1.60 (0.96)
1.61 (0.94)
1.62 (0.95)
1.60 (0.95)
PIG
---
1.57 (0.96)
1.33 (0.92)
1.55 (0.96)
1.49 (0.96)
1.32 (0.73)
1.13 (0.42)
1.56 (0.88)
1.35 (0.75)
ZINB
1.57 (0.94)
1.58 (0.92)
1.55 (0.90)
1.60 (0.91)
Poisson Hurdle
2.09 (0.97)
2.35 (0.97)
1.36 (0.75)
NB Hurdle
2.11 (0.97)
1.64 (0.82)
1.57 (0.93)
2.38 (0.97)
1.67 (0.94)
Poisson mean estimate is consistent across the varying distributions
However, coverage probability for Poisson model in the presence of overdispersion is reduced
Poisson model had consistently produced estimated standard errors that were too small
PIG model is comparable to NB but failed to converge for simulated Poisson process distributions
*Low dispersion (k=0.25) **High dispersion (k=0.05)

5. Zero-Count, Zero Truncated Poisson Zero-Count, Zero Truncated NB*
Simulation Results
Mean estimate and coverage probabilities are reported in the table below:
Important to consider if zeroes are from a single “structural” source and if more flexible models are needed to handle zero counts
If the true underlying distribution follows a zero-count, zero truncated Poisson we find that the Poisson and NB hurdle models are comparable in estimation and coverage
If the true underlying distribution follows a zero-count, zero truncated NB we find that the Poisson hurdle model performs poorly in estimation
True Distribution (β=1.60)
Zero-Count, Zero Truncated Poisson
Zero-Count, Zero Truncated NB*
Poisson
0.30 (0)
0.66 (0) NB
PIG
---
0.61 (0)
ZIP
0.79 (0)
ZINB
0.68 (0)
Poisson Hurdle
1.63 (0.95)
1.27 (0.27)
NB Hurdle
1.64 (0.96)
1.60 (0.94)
Important to consider if zeroes are from a single “structural” source and if more flexible models are needed to handle zero counts
*Low dispersion (k=0.25)

6. Application To AN RCT
Cardiothoracic Surgical Network (CTSN) Trial on Intramyocardial Injection of Mesenchymal Precursor Cells (MPCs) among Left Ventricular Assist Device (LVAD) recipients with advanced heart failure
Patients (N=159) were randomized to receive injections of MPCs or placebo and adverse events were reported at 6 months
Population of patients were more likely to experience certain type of AEs such as bleeding, infections, cardiac arrhythmias
Model estimates, fit, and performance were assessed using 1000 repeated sub-sampling and cross-validation
Poisson AIC was consistently high in the presence of specific AEs with overdispersion
In some instances, robust or sandwich estimators can be used to adjust SEs
Poisson
Distribution of MSE for Overall Bleeding
Poisson
Distribution of AIC for Overall Bleeding

7. Conclusion and Ongoing Research
Explored extended count models such as PIG, zero-Inflated, and hurdle with applications to an RCT
Prior studies have shown that the Poisson models, including ZIP and Poisson hurdle, have poorer coverage probability in the presence of underlying overdispersion
Given the potential cost of misspecification, careful consideration is needed for a “One-Model Fits All” approach and the functional form of the count responses
Further research is needed on incorporating a composite effect size across multiple AE models versus modeling the total number of AEs
Acknowledgements
Jessica Overbey, MS and Emilia Bagiella, PhD for their thoughtful insights and helpful review of the abstract and presentation.

8. References
Hilbe, J. (2014). Modeling Count Data. Cambridge: Cambridge University Press. doi: /CBO
Yau TM, Pagani FD, Mancini DM, et al. Intramyocardial Injection of Mesenchymal Precursor Cells and Successful Temporary Weaning From Left Ventricular Assist Device Support in Patients With Advanced Heart Failure: A Randomized Clinical Trial. JAMA. 2019;321(12):1176–1186. doi: /jama
Horton NJ, Kim E, Saitz R. A cautionary note regarding count models of alcohol consumption in randomized controlled trials. BMC Med Res Methodol. 2007;7:9. Published 2007 Feb 15. doi: /
Hu MC, Pavlicova M, Nunes EV. Zero-inflated and hurdle models of count data with extra zeros: examples from an HIV-risk reduction intervention trial. Am J Drug Alcohol Abuse. 2011;37(5):367–375. doi: /
Cameron AC, Trivedi PK. Regression analysis of count data. Cambridge, UK: Cambridge University Press; 1998.