Term 4, 2006BIO656--Multilevel Models 1 Midterm Open “book” and notes; closed mouth minutes to read carefully and answer completely  60 minutes to think 4 problems, with possible topics: –Some debriefing on the role of MLMs –Something on linear MLMs variances, etc. – MLMs and shrinkage –Some basic computations and interpretations of logit-linear models

Term 4, 2006BIO656--Multilevel Models 2 PART 6 PROFILING, RANKING “League Tables”

Term 4, 2006BIO656--Multilevel Models 3

Term 4, 2006BIO656--Multilevel Models 4 RANKING IN THE NEWS

Term 4, 2006BIO656--Multilevel Models 5 LETTERMAN’S TOP 10 LIST

Term 4, 2006BIO656--Multilevel Models 6 NEW YORK’S MOST DEADLY CARDIAC SURGEONS!!!!

Term 4, 2006BIO656--Multilevel Models 7

Term 4, 2006BIO656--Multilevel Models 8 THE LEADING SPH IS HARVARD

Term 4, 2006BIO656--Multilevel Models 9 HOPKINS IS THE LEADING SPH!!!

Term 4, 2006BIO656--Multilevel Models 10 PROFILING (League Tables) The process of comparing “units” on an outcome measure with relative or normative standards –Quality of care, use of services, cost –Educational quality –Disease rates in small areas –Gene expression Developing and implementing performance indices to compare physicians, hospitals, schools, teachers, genes,

Term 4, 2006BIO656--Multilevel Models 11 PROFILING OBJECTIVES PROFILING OBJECTIVES (in health services) Estimate and compare provider-specific performance measures: –Utilization/cost –Process measures –Clinical outcomes –Patient satisfaction/QoL Compare using a normative (external) or a relative (internal) standard

Term 4, 2006BIO656--Multilevel Models 12

Term 4, 2006BIO656--Multilevel Models 13 RANKING IS EASY Just compute estimates & order them

Term 4, 2006BIO656--Multilevel Models 14 MLE ESTIMATED SMRs

Term 4, 2006BIO656--Multilevel Models 15 RANKING IS DIFFICULT Need to trade-off the estimates and uncertainties

Term 4, 2006BIO656--Multilevel Models 16 MLE ESTIMATED SMRs & 95% CIs

Term 4, 2006BIO656--Multilevel Models 17 Sampling variability & Systematic variability Systematic variability Variability among physicians/hospitals that might be explained by hospital-specific characteristics Sampling variability Statistical uncertainty of physician/hospital-specific performance measures Use MLMs that Incorporate patient, physician and hospital-level characteristics Capture all important uncertainties Produce appropriate statistical summaries

Term 4, 2006BIO656--Multilevel Models 18 Statistical Challenges Need a valid method of adjusting for case mix and other features Patient, physician and hospital characteristics –But, beware of over adjustment Need a valid model for stochastic properties Account for variation at all levels Account for within-hospital, within-patient correlations Need to Adjust for systematic variation Estimate and account for statistical variation

Term 4, 2006BIO656--Multilevel Models 19 PROPER USE OF STATISTICAL SUMMARIES The challenge Differences in standard errors of hospital-specific estimates invalidate direct comparisons In any case, large SEs make comparisons imprecise Consequence Even after valid case mix adjustment, differences in directly estimated performance are due, in part, to sampling variability (Partial) Solution, use: Shrinkage estimates to balance and reduce variability Goal-specific estimates to hit the right target

Term 4, 2006BIO656--Multilevel Models 20 Comparing performance measures Ranks/percentiles, of: Direct estimates (MLEs) Shrunken estimates (BLUPs, Posterior Means) Z-scores testing H 0 that a unit is just like others Optimal (best) ranks or percentiles Other measures Probability of a large difference between unit-specific “true” and H 0 -generated event rates Probability of “excess mortality” –For the “typical patient, on average or for a specific patient type Z-score/P-value declarations....

Term 4, 2006BIO656--Multilevel Models 21

Term 4, 2006BIO656--Multilevel Models 22 USRDS

Term 4, 2006BIO656--Multilevel Models 23 USRDS

Term 4, 2006BIO656--Multilevel Models 24

Term 4, 2006BIO656--Multilevel Models 25

Term 4, 2006BIO656--Multilevel Models 26

Term 4, 2006BIO656--Multilevel Models 27

Term 4, 2006BIO656--Multilevel Models 28 MLE ESTIMATED SMRs & CIs

Term 4, 2006BIO656--Multilevel Models 29 Poisson-Normal Model Poisson-Normal Model (N, Y[k], emort[k]) are inputs model { {prec~dgamma( , ) for (k in 1:N) { 0 logsmr[k]~dnorm(0,prec) smr[k]<-exp(logsmr[k]) rate[k]<-emort[k]*smr[k] Y[k] ~ dpois(rate[k]) } Monitor the SMR[k]

Term 4, 2006BIO656--Multilevel Models 30 SE MLE PM MLE, SE & POSTERIOR MEAN SMRs (using a log-normal/Poisson model)

Term 4, 2006BIO656--Multilevel Models 31 Posterior Mean: estimated SMRs & CIs using a log-normal/Poisson model (original scale)

Term 4, 2006BIO656--Multilevel Models 32 Posterior Mean: estimated SMRs & CIs using a Gamma/Poisson model (expanded scale)

Term 4, 2006BIO656--Multilevel Models 33 Estimated relative, physician-specific visit rate and 95% CI Adjusted for patient demographic and case-mix (1.0 is the “typical” rate) Caterpillar Plot (Hofer et al. JAMA 1999)

Term 4, 2006BIO656--Multilevel Models 34 Amount that physician-specific, laboratory costs for diabetic patients deviates from the mean for all physicians [$/(pt. yr.)] Lines show the path from the direct estimate (the MLE) to the shrunken estimate (Hofer et al JAMA 1999) DIRECTADJUSTED

Term 4, 2006BIO656--Multilevel Models 35 Example using BUGS for hospital performance ranking

Term 4, 2006BIO656--Multilevel Models 36

Term 4, 2006BIO656--Multilevel Models 37 BUGS Model specification model { for k in 1:K { b[k]~dnorm(0, prec) r[k]~dbin(p[k], n[k]) logit(p[k]) <- mu + b[k] } pop.mean<-exp(mu + bb)/(1+exp(mu + bb)) mu~dnorm(0, 1E-6) prec~dgamma(.0001,.0001) tausq<-1/prec add~dnorm(0, prec) bb<- mu + add } Monitor the p[k] and ask for ranks

Term 4, 2006BIO656--Multilevel Models 38 Summary Statistics

Term 4, 2006BIO656--Multilevel Models 39 Posterior distributions of the ranks

Term 4, 2006BIO656--Multilevel Models 40

Term 4, 2006BIO656--Multilevel Models 41

Term 4, 2006BIO656--Multilevel Models 42

Term 4, 2006BIO656--Multilevel Models 43 X = (Posterior Mean-Based Ranks) – (Optimal Ranks) LOS  = = = =

Term 4, 2006BIO656--Multilevel Models 44 LOS

Term 4, 2006BIO656--Multilevel Models 45 Relations among percentiling methods 1998 USRDS Percentiles

Term 4, 2006BIO656--Multilevel Models 46 BACK TO THE USRDS, SMR EXAMPLE

Term 4, 2006BIO656--Multilevel Models 47

Term 4, 2006BIO656--Multilevel Models 48

Term 4, 2006BIO656--Multilevel Models 49 False detection and non-detection

Term 4, 2006BIO656--Multilevel Models 50

Term 4, 2006BIO656--Multilevel Models 51 K is large and we can use a completely non-parametric prior

Term 4, 2006BIO656--Multilevel Models 52  = (1-B) =  2 /(  2 +  2 ) = ICC

Term 4, 2006BIO656--Multilevel Models 53 Probability of being in the upper 10% as a function of true percentile  = intra-class correlation

Term 4, 2006BIO656--Multilevel Models 54