Chaoran Hu1,4, Xiao Tan2,4, Qing Pan3, Yong Ma4, Jaejoon Song4 Random Forests for Exploring Factors Driving Opioid Prescribing in National Outpatient Health Care Data Using Complex Survey Design Chaoran Hu1,4, Xiao Tan2,4, Qing Pan3, Yong Ma4, Jaejoon Song4 1 University of Connecticut, Department of Statistics 2 George Mason University, Department of Statistics 3 George Washington University, Department of Statistics 4 U.S. Food and Drug Administration, Center for Drug Evaluation and Research Joint Statistical Meeting, 2019

Disclaimer This presentation reflects the views of the author and should not be construed to represent FDA’s views or policies.

Background and study goals The opioid crisis Reduce unnecessary prescription is a key Need to understand opioid prescription pattern and identify important predictors of opioid prescription The NAMCS survey Nation-wide complex survey conducted by CDC Penalized logistic regression (PLR) with complex survey data LASSO with weighted logistic regression Random Forest (RF) with complex survey data Weighted RF Goals Compare the results of PLR with RF in complex survey data Evaluate the performances using cross validation www.fda.gov

Data description Stratum: Geographical regions Cluster: Physicians 2016 national ambulatory medical care survey data (NAMCS)1 Data collected by using complex survey structure simplified to stratified 2 stage sampling (see figure on the right) 10031 observations Response variable: opioid prescription (binary) Covariates: 190 deemed relevant, after removing highly correlated covariates, final covariates used is 177: number of medication other than opioid, physician specialty, usage of tobacco, total number of chronic conditions and others. Sampling weights: at patient level ________ 1. https://www.cdc.gov/nchs/ahcd/index.htm Stratum: Geographical regions Cluster: Physicians Weight: Patients www.fda.gov

Comparison of LASSO and Random Forest (RF) Y axis lists variables sorted by importance from RF model (bottom the most important) X axis shows increasing λ in the LASSO model and variables with shaded area remaining in the model A perfect match between LASSO and RF would show a shaded area covering the lower 45 degree region www.fda.gov λ in the LASSO model

Comparison of LASSO and Random Forest via cross-validation Overall classification error rate ROC curves Cross-validation: 2/3 of the original data was used to fit a LASSO or RF model and 1/3 used for validation and this was repeated 500 times Overall, Random Forest preforms better than LASSO (RF AUC at 0.83 vs. LASSO AUC at 0.81) www.fda.gov

Conclusions Random Forest and LASSO produced different yet similar results. Differences can be explained by the following factors Functional form of the continuous variables. Inclusion of the interaction terms. Handling highly correlated covariates. RF performs better than LASSO in terms of AUC, sensitivity and overall classification rate at cutoff 0.5. RF can be used as a tool to build a better LASSO regression model, or vice versa. Alternatively, a Super Learner approach can be used to join forces1. _______________________ 1 Van der Laan MJ, Polley EC, Hubbard AE. Super learner. Stat Appl Genet Mol Biol. 2007;6:Article25 www.fda.gov

Acknowledgement This study was made possible by funding from FDA’s Regulatory Science and Review Enhancement Program (RSR, FY 2019). This project was also supported in part by an appointment to the Oak Ridge Institute for Science and Education (ORISE) Research Participation Program at FDA/CDER, administered by ORISE through an interagency agreement between the U.S. Department of Energy and FDA/CDER