Public Health Physician, Lecturer Critical Care Medicine, Statistics in Medicine Research Article Received 18 June 2012, Accepted 30 January 2013 Published online in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/sim.5779 Evaluating the performance of Australian and New Zealand intensive care units in 2009 and 2010 J. Kasza, J. L. Moran, P. J. Solomon for the ANZICS Centre for Outcome and Resource Evaluation (CORE) of the Australian and New Zealand Intensive Care Society (ANZICS) Presented at the Flinders Centre for Epidemiology and Biostatistics (FCEB) Journal Club – 9th May 2013 by Dr Wayne Clapton Public Health Physician, Lecturer Critical Care Medicine, Flinders University 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton SUMMARY-1 Australian and New Zealand Intensive Care Society (ANZICS) Adult Patient Database (APD) Collects Individual Admissions Data - Aust & New Zealand Intensive Care Units (ICU’s) Used for monitoring and comparing performance of ICU’s Identifies unusual ICU’s via Standardised Mortality Ratios with 95% or 99% confidence intervals or Funnel Plots with 95% or 99% prediction limits. Theoretical and computational challenges arise from this approach. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton SUMMARY-2 Specific statistical issues often neglected in ICU comparison studies: Expected number of deaths must be estimated appropriately ICU casemix adjustment must be adequate Must account fully for statistical variation Must adjust appropriately for multiple comparisons 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton SUMMARY-3 Authors’ approach: Fit a random coefficient hierarchical logistical model for the in-hospital death of each patient, with patients clustered within ICU’s. Anticipate that the majority of ICU’s will be estimated as performing “usually” after adjusting for important clinical covariates. Start with Ohlssen et al’s ideas and estimate an appropriate null model that ICU’s would be expected to follow (A Frequentist, rather than a Bayesian approach) 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton SUMMARY-4 Results of Authors’ methodology: Rigorously account for the statistical issues Determine if any ICU’s on the ANZICS APD have comparatively unusual performance Investigate annual ICU performance Estimate changes in individual ICU performance between 2009 and 2010 by adjusting for regression-to-the-mean. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

STANDARDISED MORTALITY RATIOS Used as performance indicators Expected number of deaths for each ICU is estimated by summing patient mortality probabilities adjusted for severity of the patient’s conditions. Despite some inclusion of other covariates, other clinically important covariates are not present. Expected number of deaths are assumed fixed when calculating SMR’s variances. Unusual ICU performers are detected when: SMR confidence intervals do not contain one SMR’s fall outside prediction limits of the Funnel Plots 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton HIERARCHICAL MODELS - 1 Standard use for comparing healthcare providers; but not always used with patients clustered within ICU’s for comparing ICU performance. Two approaches for comparing providers: Fit a random effects distribution that encompasses all variation between providers and identify those providers that have extreme random effects with respect to the model. Fit a random effects distribution that the majority of providers are expected to follow and identify any providers who have outlying random effects with respect to this model, using hypothesis testing. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton HIERARCHICAL MODELS - 2 Most ICU’s will have usual performance; but not be identical. The interest is in identifying unusually poor or good performance. Estimate a random coefficient hierarchical logistic regression model for the in-hospital death for each patient. Clustering patients within ICU’s Random effects distribution that describes mortality experience in an ICU with usual performance. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

MODEL - CASEMIX ADJUSTMENT Included in the model: Age, sex, patient severity score, patient diagnostic category, patient surgical status, patient ventilation status, source of admission to the ICU, geographical locality, ICU level, annual ICU volume, clinically meaningful interactions (see Table 2 in the supplementary information) Random intercept Random coefficient for patient severity score Patient severity score: Acute Physiology and Chronic Health Evaluation, third revision (APACHE III) – worst clinical values in the first 24 hours post-admission 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton MODEL - OTHER Log-SMR’s, rather than SMR’s, used as performance indicators to obtain confidence intervals with better coverage properties. Uncertainty in both observed and expected numbers of deaths are accounted for when calculating log-SMR variances. The multiplicity of hypothesis tests is accounted for by controlling the False Discovery Rate (FDR). Funnel plots are used to display results. Thresholds controlling the FDR are placed on the plots. Yearly performance plus change between 2009 and 2010. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

ANZICS APD & CHOICE OF RISK-ADJUSTED MODEL ANZICS APD – Individual patients’ admissions data, large, voluntary, data quality checks, extracts to CORE Study dataset exclusions: Unknown hospital mortality status and discharge date (4023 patients), Patients with an ICU stay less than 4 hours (2110), Patients less than 16 years old (3822) 10170 separate hospital readmissions ICU’s with fewer than 150 patients in 2009 or 2010 Of 139 ICU’s in 2009, 24 excluded Of 142 ICU’s in 2012, 27 excluded. Final dataset: 163,795 patients from 115 ICU’s 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

HIERARCHICAL LOGISTIC REGRESSION MODEL - 1 Mortality experience is not independent year to year (Continuity of hospital-level and ICU level factors from year to year, many of which are outside the control of the ICU.) So cannot fit separate models for 2009 and 2010. Thus models incorporate data from both years, with patients clustered within ICU’s. 30 day-after-hospital-discharge mortality would be more appropriate; but no APD linkage with death registers to enable this. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

HIERARCHICAL LOGISTIC REGRESSION MODEL - 2 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

HIERARCHICAL LOGISTIC REGRESSION MODEL - 3 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

HIERARCHICAL LOGISTIC REGRESSION MODEL - 4 Building an appropriate model for risk-adjustment: Patients split into an 80% training set and a 20% test set. Model parameters estimated using xtmelogit (Stata 12), with the Laplace approximation used in the model-building stage. Model 1: All covariates and two-way interactions; but excluded a random coefficient for APACHE III score. Model 2: Included a random coefficient for APACHE III score. Model 3: Removing interactions insignificant at the 5% level (ICU source by each of APACHE III score and surgical status; and ventilation status by each of age, sex and surgical status.) 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

HIERARCHICAL LOGISTIC REGRESSION MODEL - 5 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

HIERARCHICAL LOGISTIC REGRESSION MODEL - 6 Model 3 was chosen for risk adjustment. No interactions were insignificant at the 5% level. 110 fixed effects parameters and Three random effects parameters. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING ICU’s WITH UNUSUAL PERFORMANCE USUAL APPROACHES - 1 Usual approaches (which have problems): Estimate mortality probabilities using a scoring algorithm, e.g., APACHE III. ANZICS Centre for Outcome and Resource Evaluation (ANZICS CORE) uses this method with APACHE III-J Fit a logistic regression model similar to Equation 1, where the log odds of in-hospital patient deaths are dependent upon a severity score. Can extend to account for hierarchical nature of patient-level data, plus include covariates and a random coefficient for severity score. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING ICU’s WITH UNUSUAL PERFORMANCE USUAL APPROACHES - 2 Problems with the usual methods: Method 1: Fails to account for patient clustering within ICU’s. Scoring systems may not be calibrated for the population under study. Method 2: Adjustment for risk is inadequate, because many clinically important covariates often are not included. Both: The presence of unusual ICU’s is not accounted for in the estimation of a model describing the mortality experience of ICU’s with usual performance. A model estimated using data from both usually and unusually performing ICU’s is likely to have over-inflated random effect variance estimates, which leads to overestimation of the usual levels of variation in SMR’s. Risk of misinterpretation of confidence intervals. Expected number of deaths assumed to be fixed in SMR variance estimations for funnel plot construction. Multiple comparisons are not addressed. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 - 1 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 – 2 STAGE 1 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 – 3 STAGE 1 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 – 4 STAGE 1 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 – 5 STAGE 1 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 – 6 STAGE 1 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 STAGE 2 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 STAGE 2 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 STAGE 2 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 STAGE 3 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 STAGE 3 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 STAGE 3 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING UNUSUAL ICU’s IN APD 2009 & 2010 - STAGE 3 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING ICU’s WITH RECENT CHANGES IN PERFORMANCE 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING ICU’s WITH RECENT CHANGES IN PERFORMANCE 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING ICU’s WITH RECENT CHANGES IN PERFORMANCE 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

IDENTIFYING ICU’s WITH RECENT CHANGES IN PERFORMANCE 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton DISCUSSION - 1 Key concepts and principled statistical methods Estimating the null model for in-hospital patient mortality and adjusting for important casemix factors  Rigorously monitored 2009 & 2010 APD ICU’s. Formally compared 2009-2010 changes via regression-to-the-mean techniques. Reduced the risk of incorrectly identifying ICU’s as unusual. Reduced the risk of failing to identify ICU’s performing unusually, be the performance unusually good or unusually poor. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton DISCUSSION - 2 Suggested that random intercept captures unknown ICU-level explanatory variables (possibly related to performance). Therefore, recommend a null random-effects density. ICU’s detected by outlying log-SMR’s should be investigated further - as per CORE policy. Further years of data may be required to check performance outliers re whether this is a real effect. Results here differ from the ANZICS CORE Report – suggest more rigorous statistical methods by CORE. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton DISCUSSION - 3 Adding further interactions and/or covariates significantly adds to computer run times and may over-adjust for risk? 30-day mortality after hospital discharge would be better for survival analyses – need linkage. Comparisons for periods more than two years is of interest and would require extensions of the methods described here. Models more than two years incorporating autocorrelation and seasonal effects are of interest – develop at the risk- adjustment stage of the modelling process. 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton

COVARIATES USED IN THE MODELS 1/12/2018 FCEB Journal Club Presentation 9th May 2013 - Dr Wayne Clapton