1. Term 4, 2006BIO656--Multilevel Models 1 PART 07 Evaluating Hospital Performance

2. Term 4, 2006BIO656--Multilevel Models 2 PERFORMANCE MEASURES Patient outcomes Mortality, morbidity, satisfaction with care –30-day mortality among heart attack patients (Normand et al JAMA 1996, JASA 1997) Process Medication & test administration, costs –Laboratory costs for diabetic patients –Number of physician visits Hofer et al JAMA, 1999 –Palmer et al. (1996)

3. Term 4, 2006BIO656--Multilevel Models 3 DATA STRUCTURE Multi-level Patients nested in physicians, hospitals, HMOs,... Providers clustered by health care systems, market areas, geographic areas Covariates at different levels of aggregation: –patient, physician, hospital,... Variation in variability Statistical stability varies over physicians, hospitals,..

4. Term 4, 2006BIO656--Multilevel Models 4 MLMs are Effective Correlation at many levels Hospital practices may induce a strong correlation among patient outcomes within hospitals even after accounting for patient characteristics Structuring estimation Stabilizing noisy estimates Balancing SEs Estimating ranks and other non-standard summaries

5. Term 4, 2006BIO656--Multilevel Models 5 The Cooperative Cardiovascular Project (CCP) The Cooperative Cardiovascular Project (CCP) Abstracted medical records for patients discharged from hospitals located in Alabama, Connecticut, Iowa, and Wisconsin (June 1992  May 1993) 3,269 patients hospitalized in 122 hospitals in four US States for Acute Myocardial Infarction

6. Term 4, 2006BIO656--Multilevel Models 6 GOALS Identify “aberrant” hospitals with respect to several performance measures Report the statistical uncertainty associated with ranking of the “worst hospitals” Investigate if hospital characteristics explain variation in hospital-specific mortality rates

7. Term 4, 2006BIO656--Multilevel Models 7 DATA Outcome Mortality within 30-days of hospital admission Patient characteristics Admission severity index constructed on the basis of 34 patient attributes Hospital characteristics Urban/Rural (Non academic)/(versus academic) Number of beds

8. Term 4, 2006BIO656--Multilevel Models 8 Why adjust for case mix? Why adjust for case mix? (patient characteristics) Irrespective of quality of care, older/sicker patients with multiple diseases have increased need of health care services and poorer health outcomes Without adjustment, physicians/hospitals who treat relatively more of these patients will appear to provide more expensive and lower quality care than those who see relatively younger/healthier patients  If there is inadequate case mix adjustment, evaluations will be unfair But, need to avoid over adjusting

9. Term 4, 2006BIO656--Multilevel Models 9 Case-mix Adjustment Compute hospital-specific, expected mortality by: 1.estimating a patient-level mortality model using all hospitals 2. averaging the model-produced probabilities for all patients within a hospital Hospitals with “higher-than-expected” mortality rates can be flagged as institutions with potential quality problems, but need to account for uncertainty Need to be careful, if also adjusting for hospital characteristics –May adjust away the important signal

10. Term 4, 2006BIO656--Multilevel Models 10 (as we know, very poor approach) Wrong SEs Test-based

11. Term 4, 2006BIO656--Multilevel Models 11 Hospital Profiling of Mortality Rates Acute Myocardial Infarction Patients Hospital Profiling of Mortality Rates Acute Myocardial Infarction Patients (Normand et al. JAMA 1996, JASA 1997)

12. Term 4, 2006BIO656--Multilevel Models 12 Hierarchical logistic regression I: Patient within-provider Patient-level logistic regression model with random intercept & slope II: Between-provider Hospital-specific random effects are regressed on hospital-specific characteristics –Explicit regression

13. Term 4, 2006BIO656--Multilevel Models 13 Admission severity index Admission severity index (Normand et al. 1997 JASA)

14. Term 4, 2006BIO656--Multilevel Models 14  0 +  1 (sev ij – sevbar) 00 11 sevbar

15. Term 4, 2006BIO656--Multilevel Models 15 we use b 0i + b 1i (...)

16. Term 4, 2006BIO656--Multilevel Models 16 Interpretation of parameters is different for the two levels b 0i =  * 00 + N(..), etc.

17. Term 4, 2006BIO656--Multilevel Models 17 RESULTS Estimates of regression coefficients under three models: –Random intercept only –Random intercept and random slope –Random intercept, random slope, and hospital covariates Hospital performance measures

18. Term 4, 2006BIO656--Multilevel Models 18 Normand et al. JASA 1997

19. Term 4, 2006BIO656--Multilevel Models 19 30-DAY MORTALITY 30-DAY MORTALITY 2.5 th and 97.5 th percentiles for a patient of average admission severity Exchangeable model Random intercept and slope, no hospital covariates log(odds): (-1.87,-1.56) probability,scale: (0.13, 0.17) Covariate (non-exchangeable) model Random intercept and slope, with hospital covariates Patient treated in a large, urban academic hospital log(odds): (-2.15,-1.45) probability scale: (0.10,0.19)

20. Term 4, 2006BIO656--Multilevel Models 20 Effect of hospital characteristics on baseline log-odds of 30-day mortality For an average patient, rural hospitals have a higher odds ratio than urban hospitals –Indicates between-hospital differences in the baseline mortality rates –Case-mix adjustment may be able to remove some of this difference

21. Term 4, 2006BIO656--Multilevel Models 21 Estimates of Stage-II regression coefficients Intercepts

22. Term 4, 2006BIO656--Multilevel Models 22 Effect of hospital characteristics on association between severity and mortality Effect of hospital characteristics on association between severity and mortality (slopes) The association between severity and mortality is modified by hospital size Medium-sized hospitals have smaller severity/mortality associations than large hospitals –Indicates that the effect of clinical burden (patient severity) on mortality differs across hospitals

23. Term 4, 2006BIO656--Multilevel Models 23 Estimates of Stage II regression coefficients Slopes

24. Term 4, 2006BIO656--Multilevel Models 24 Homework is on front table

25. Term 4, 2006BIO656--Multilevel Models 25 Observed and risk-adjusted hospital mortality rates Urban Hospitals Histogram displays (observed – adjusted)

26. Term 4, 2006BIO656--Multilevel Models 26 Observed and risk-adjusted hospital mortality rates Rural Hospitals Histogram displays (observed – adjusted) Substantial adjustment for severity

27. Term 4, 2006BIO656--Multilevel Models 27 FINDINGS There is substantial adjustment for admission severity Generally, urban hospitals are adjusted less than rural There is less variability in observed or adjusted estimated rates for urban hospitals than for rural hospitals Can you explain why?

28. Term 4, 2006BIO656--Multilevel Models 28 Normand et al. JASA 1997

29. Term 4, 2006BIO656--Multilevel Models 29 Average the probabilities Don’t average the covariates

30. Term 4, 2006BIO656--Multilevel Models 30 k denotes a draw from the posterior

31. Term 4, 2006BIO656--Multilevel Models 31 Plug in the average covariate Keep the hospital variation

32. Term 4, 2006BIO656--Multilevel Models 32

33. Term 4, 2006BIO656--Multilevel Models 33 Comparing measures of hospital performance Three measures of hospital performance Probability of a large difference between adjusted and standardized mortality rates Probability of excess mortality for the average patient Z-score

34. Term 4, 2006BIO656--Multilevel Models 34 Hospital Rankings: Normand et al 1997 JASA

35. Term 4, 2006BIO656--Multilevel Models 35 Hospital Ranks There was moderate disagreement among the criteria for classifying hospitals as “aberrant” Nevertheless, hospital 1 is ranked worst It is rural, medium sized non-academic with an observed mortality rate of 35%, and adjusted rate of 28%

36. Term 4, 2006BIO656--Multilevel Models 36 Adjusting for hospital-level charateristics Changes the comparison group in “as compared to what?” All hospitals (unadjusted at hospital level) Hospitals of a similar size, urbanicity,... Percent of physicians who are board certified Hospitals with a similar death rate  Variance reduction and goodness of fit should not be the primary considerations “As compared to what?” must dominate

37. Term 4, 2006BIO656--Multilevel Models 37 Discussion Profiling medical providers is multi-faced and data intensive process with substantial implications for health care practice, management, and policy Major issues include data quality and availability, choice of performance measures, formulation of statistical models (including adjustments), reporting results The ranking approaches and summaries used by Normand and colleagues are very good, but some improvement is possible

38. Term 4, 2006BIO656--Multilevel Models 38 Multi-level models address key technical & conceptual profiling issues, including Adjusting for patient severity Accounting for within-provider correlations Accounting for differential sample sizes at all levels Stabilize estimates Structure ranking and other, derived comparisons