Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance

Term 4, 2006BIO656--Multilevel Models 2 PERFORMANCE MEASURES Patient outcomes Mortality, morbidity, satisfaction with care –30-day mortality among heart attack patients (Normand et al JAMA 1996, JASA 1997) Process Medication & test administration, costs –Laboratory costs for diabetic patients –Number of physician visits Hofer et al JAMA, 1999 –Palmer et al. (1996)

Term 4, 2006BIO656--Multilevel Models 3 DATA STRUCTURE Multi-level Patients nested in physicians, hospitals, HMOs,... Providers clustered by health care systems, market areas, geographic areas Covariates at different levels of aggregation: –patient, physician, hospital,... Variation in variability Statistical stability varies over physicians, hospitals,..

Term 4, 2006BIO656--Multilevel Models 4 MLMs are Effective Correlation at many levels Hospital practices may induce a strong correlation among patient outcomes within hospitals even after accounting for patient characteristics Structuring estimation Stabilizing noisy estimates Balancing SEs Estimating ranks and other non-standard summaries

Term 4, 2006BIO656--Multilevel Models 5 The Cooperative Cardiovascular Project (CCP) The Cooperative Cardiovascular Project (CCP) Abstracted medical records for patients discharged from hospitals located in Alabama, Connecticut, Iowa, and Wisconsin (June 1992  May 1993) 3,269 patients hospitalized in 122 hospitals in four US States for Acute Myocardial Infarction

Term 4, 2006BIO656--Multilevel Models 6 GOALS Identify “aberrant” hospitals with respect to several performance measures Report the statistical uncertainty associated with ranking of the “worst hospitals” Investigate if hospital characteristics explain variation in hospital-specific mortality rates

Term 4, 2006BIO656--Multilevel Models 7 DATA Outcome Mortality within 30-days of hospital admission Patient characteristics Admission severity index constructed on the basis of 34 patient attributes Hospital characteristics Urban/Rural (Non academic)/(versus academic) Number of beds

Term 4, 2006BIO656--Multilevel Models 8 Why adjust for case mix? Why adjust for case mix? (patient characteristics) Irrespective of quality of care, older/sicker patients with multiple diseases have increased need of health care services and poorer health outcomes Without adjustment, physicians/hospitals who treat relatively more of these patients will appear to provide more expensive and lower quality care than those who see relatively younger/healthier patients  If there is inadequate case mix adjustment, evaluations will be unfair But, need to avoid over adjusting

Term 4, 2006BIO656--Multilevel Models 9 Case-mix Adjustment Compute hospital-specific, expected mortality by: 1.estimating a patient-level mortality model using all hospitals 2. averaging the model-produced probabilities for all patients within a hospital Hospitals with “higher-than-expected” mortality rates can be flagged as institutions with potential quality problems, but need to account for uncertainty Need to be careful, if also adjusting for hospital characteristics –May adjust away the important signal

Term 4, 2006BIO656--Multilevel Models 10 (as we know, very poor approach) Wrong SEs Test-based

Term 4, 2006BIO656--Multilevel Models 11 Hospital Profiling of Mortality Rates Acute Myocardial Infarction Patients Hospital Profiling of Mortality Rates Acute Myocardial Infarction Patients (Normand et al. JAMA 1996, JASA 1997)

Term 4, 2006BIO656--Multilevel Models 12 Hierarchical logistic regression I: Patient within-provider Patient-level logistic regression model with random intercept & slope II: Between-provider Hospital-specific random effects are regressed on hospital-specific characteristics –Explicit regression

Term 4, 2006BIO656--Multilevel Models 13 Admission severity index Admission severity index (Normand et al JASA)

Term 4, 2006BIO656--Multilevel Models 14  0 +  1 (sev ij – sevbar) 00 11 sevbar

Term 4, 2006BIO656--Multilevel Models 15 we use b 0i + b 1i (...)

Term 4, 2006BIO656--Multilevel Models 16 Interpretation of parameters is different for the two levels b 0i =  * 00 + N(..), etc.

Term 4, 2006BIO656--Multilevel Models 17 RESULTS Estimates of regression coefficients under three models: –Random intercept only –Random intercept and random slope –Random intercept, random slope, and hospital covariates Hospital performance measures

Term 4, 2006BIO656--Multilevel Models 18 Normand et al. JASA 1997

Term 4, 2006BIO656--Multilevel Models DAY MORTALITY 30-DAY MORTALITY 2.5 th and 97.5 th percentiles for a patient of average admission severity Exchangeable model Random intercept and slope, no hospital covariates log(odds): (-1.87,-1.56) probability,scale: (0.13, 0.17) Covariate (non-exchangeable) model Random intercept and slope, with hospital covariates Patient treated in a large, urban academic hospital log(odds): (-2.15,-1.45) probability scale: (0.10,0.19)

Term 4, 2006BIO656--Multilevel Models 20 Effect of hospital characteristics on baseline log-odds of 30-day mortality For an average patient, rural hospitals have a higher odds ratio than urban hospitals –Indicates between-hospital differences in the baseline mortality rates –Case-mix adjustment may be able to remove some of this difference

Term 4, 2006BIO656--Multilevel Models 21 Estimates of Stage-II regression coefficients Intercepts

Term 4, 2006BIO656--Multilevel Models 22 Effect of hospital characteristics on association between severity and mortality Effect of hospital characteristics on association between severity and mortality (slopes) The association between severity and mortality is modified by hospital size Medium-sized hospitals have smaller severity/mortality associations than large hospitals –Indicates that the effect of clinical burden (patient severity) on mortality differs across hospitals

Term 4, 2006BIO656--Multilevel Models 23 Estimates of Stage II regression coefficients Slopes

Term 4, 2006BIO656--Multilevel Models 24 Homework is on front table

Term 4, 2006BIO656--Multilevel Models 25 Observed and risk-adjusted hospital mortality rates Urban Hospitals Histogram displays (observed – adjusted)

Term 4, 2006BIO656--Multilevel Models 26 Observed and risk-adjusted hospital mortality rates Rural Hospitals Histogram displays (observed – adjusted) Substantial adjustment for severity

Term 4, 2006BIO656--Multilevel Models 27 FINDINGS There is substantial adjustment for admission severity Generally, urban hospitals are adjusted less than rural There is less variability in observed or adjusted estimated rates for urban hospitals than for rural hospitals Can you explain why?

Term 4, 2006BIO656--Multilevel Models 28 Normand et al. JASA 1997

Term 4, 2006BIO656--Multilevel Models 29 Average the probabilities Don’t average the covariates

Term 4, 2006BIO656--Multilevel Models 30 k denotes a draw from the posterior

Term 4, 2006BIO656--Multilevel Models 31 Plug in the average covariate Keep the hospital variation

Term 4, 2006BIO656--Multilevel Models 32

Term 4, 2006BIO656--Multilevel Models 33 Comparing measures of hospital performance Three measures of hospital performance Probability of a large difference between adjusted and standardized mortality rates Probability of excess mortality for the average patient Z-score

Term 4, 2006BIO656--Multilevel Models 34 Hospital Rankings: Normand et al 1997 JASA

Term 4, 2006BIO656--Multilevel Models 35 Hospital Ranks There was moderate disagreement among the criteria for classifying hospitals as “aberrant” Nevertheless, hospital 1 is ranked worst It is rural, medium sized non-academic with an observed mortality rate of 35%, and adjusted rate of 28%

Term 4, 2006BIO656--Multilevel Models 36 Adjusting for hospital-level charateristics Changes the comparison group in “as compared to what?” All hospitals (unadjusted at hospital level) Hospitals of a similar size, urbanicity,... Percent of physicians who are board certified Hospitals with a similar death rate  Variance reduction and goodness of fit should not be the primary considerations “As compared to what?” must dominate

Term 4, 2006BIO656--Multilevel Models 37 Discussion Profiling medical providers is multi-faced and data intensive process with substantial implications for health care practice, management, and policy Major issues include data quality and availability, choice of performance measures, formulation of statistical models (including adjustments), reporting results The ranking approaches and summaries used by Normand and colleagues are very good, but some improvement is possible

Term 4, 2006BIO656--Multilevel Models 38 Multi-level models address key technical & conceptual profiling issues, including Adjusting for patient severity Accounting for within-provider correlations Accounting for differential sample sizes at all levels Stabilize estimates Structure ranking and other, derived comparisons