1. Islamabad - 2008 1 Measuring performance of ICUs Does it help to improve? Bertrand Guidet Hôpital St Antoine Paris, France Réanimation Médicale & INSERM U707

2. Islamabad - 2008 2 Performance indicators Mortality Activity Efficiency/cost Structure/processes

3. Islamabad - 2008 3 Mortality

4. Islamabad - 2008 4 Interpretation of mortality data Standardized mortality ratio : SMR = Observed mortality/predicted mortality In hospital mortality is estimated with severity scores: –SAPS 2 or SAPS 3 –APACHE II or PACHE III –MPM If SMR < 1 : « good performance » The observed mortality is lower than the expected mortality

5. Islamabad - 2008 5 Observed/predicted mortality (SMR) according to the origin of the patients CUB-REA data base : year 2005 (29 ICUs)

6. Islamabad - 2008 6 Mortality and SMR according to diagnosis CUB-REA 2005

7. Islamabad - 2008 7 Ranking of ICUs with adjusted SMR Aegerter P,… Guidet B SAPS 2 revisited (ICM 2005, 31 : 416-423)

8. Islamabad - 2008 8 Ranking change after adjustment Model B : Age Mode of entry Comorbidities

9. Islamabad - 2008 9 Standardized mortality in ICU and in hospital

10. Islamabad - 2008 10 Standardized hospital mortality for specific diagnosis Diagnosis ARF/COPD ARDSSeptic Shock SMR 0.3 0.8 1.3 1.8 2.3 2.8 3.3 3.8 ICUs 01234567891011121314151617181920212223242526

11. Islamabad - 2008 11 Activity

12. Islamabad - 2008 12 Main data of the CUB-Rea annual report (37 ICUs, Paris area) For the whole database Per ICU with identification of each ICU Mean (median, sd, range) Comparison with previous years

13. Islamabad - 2008 13 Description N patientsAgeLOS Mean Median 56.0 y7.5 days4.0 days Global characteristics of the patients Severity & mortality SAPS 2ICU mortalityHospital mortalitySMR median% of patients 34.018.0%23.1%0.79 mean 38.4 Median 56.7 y 540 per ICU

14. Islamabad - 2008 14 Workload and organ support

15. Islamabad - 2008 15 Rationale for reporting volume of activity

16. Islamabad - 2008 16 VOLUME and PERFORMANCE Halm et al, Ann Int Med, 2002. Is volume related to outcome in health care ? A systematic review and methodological critique of the literature –Review of 135 studies with 27 procedures –There is a significant statistical association between volume and outcome in 71 % studies on hospital volume 69 % studies on physician activities

17. Islamabad - 2008 17 Hospital volume and surgical mortality in the United States Birkmeyer et al, N Engl J Med, 2002. –Mortality decreased as volume increased for all 14 types of procedures pancreatic resection :12%(16.3%vs. 3.8%) carotid endarterectomy : 1.6% (1.7% vs 1.5%).

18. Islamabad - 2008 18 Surgeon Volume and Operative Mortality in the United States JD. Birkmeyer, NEJM 2003, 349 :2117 –For many procedures, the observed associations between hospital volume and operative mortality are largely mediated by surgeon volume. –Patients can often improve their chances of survival substantially, even at high-volume hospitals, by selecting surgeons who perform the operations frequently

19. Islamabad - 2008 19 Hospital Volume and the outcomes of mechanical ventilation Kahn, NEJM 2006, 355:41-50 20 241 non surgical patients 37 hospitals From 2002 through 2003

20. Islamabad - 2008 20 Risk-adjusted mortality Kahn, NEJM 2006, 355:41-50

21. Islamabad - 2008 21 44,436 patients receiving mechanical ventilation on admission to 38 ICUs 894 patients (2%) cared for at 5 ICUswith missing data 160 patients (0.3 %) with hemato- oncologic disease 1635 patients (3,7%) with drug- induced coma 41,747 patients receiving mechanical ventilation on admission to 33 ICUs 44,436 patients receiving mechanical ventilation on admission to 38 ICUs 894 patients (2%) cared for at 5 I CUswith missing data 160 patients (0.3 %) with hemato - oncologic disease 1635 patients (3,7%) with drug - induced coma 41,747 patients receiving mechanical ventilation on admission to 33 ICUs CUB-REA data base 8 years

22. Islamabad - 2008 22 Characteristics of patients Quartile 1 Quartile 2 Quartile 3 Quartile 4 p Number4,3498,17910,49518,724 - Age6060.257.9 <0.001 Female %394040.2410.06 SAPS II51.65753.353.8<0.001

23. Islamabad - 2008 23 Mortality Quartile 1 Quartile 2 Quartile 3 Quartile 4 p ICU Mortality 45.845.940.238.2 <0.0001 Hospital Mortality 4848.440.940.7 ICU Length of stay 12.1 1210.2

24. Islamabad - 2008 24 Case-Volume and Mortality in Hematological Patients with Acute Respiratory Failure. Eur Respir J. 2008; 32 (in press) A case volume increase of one admission per year led to a significant mortality reduction with an odd ratio of 0.98 (95% CI : 0.97 – 0.99)

25. Islamabad - 2008 25 Economic performance

26. Islamabad - 2008 26 Crude economic performance Direct medical costs Study on 21 French ICUs Yearly total Cost per ICU Cost/patient /day Cost/stayCost/ bed/day Mean3 665 885 €985 €5 990 €702 € Std deviation1 109 126 €257 €2 740 €179 € Highest 1 912 812 €670 €3 627 €481 € Lowest 6 425 362 €1 537 €12 599 €1 114 €  There is a need for adjustment to take into account the case-mix

27. Islamabad - 2008 27 How to detect «a dangerous ICU» ? The case of Dr Shipman Dr Shipman was a general practioner who murdered some of his patients from 1977 to 1997 : –180 women and 55 men aged 65 years or over. –He was arrested in 1998 Could this 20-year delay been reduced with an alarm - alert system ?

28. Islamabad - 2008 28 ALARM - ALERT Tekkis et al, BMJ, 2003, Mortality control charts for comparing performance of surgical units: validation study using hospital mortality data. –A two level hierarchical logistic regression model was used to adjust each unit’s operative mortality 1: case-mix : patient associated factor; 2: hospital associated factor

29. Islamabad - 2008 29

30. Islamabad - 2008 30

31. Islamabad - 2008 31 ALARM - ALERT Spiegelhalter et al, Int J Qual Health Care, 2003, Risk-adjusted sequential probability ratio tests : applications to Bristol, Shipman, and adult cardiac surgery. –Cumulative excess mortality in Bristol for cardiac surgery HES : hospital episode statistics CSR : cardiac surgery register

32. Islamabad - 2008 32 Global quality improvement process Benchmarking is the first step of the quality improvement process. Adjustment technique : two level hierarchical logistic regression model to take into account patients variables (case-mix) and hospital/unit characteristics. Once discrepancies between an ICU and the comparator are identified, objectives for improvement should be set.

33. Islamabad - 2008 33 1 2 Goal Identification of the indicator  Data collection (who, how, …)  Data analysis and presentation  Goal (level, time) Observations : 1- …………… 2- …………… Reference Who is in charge ? Who controls ? Time scaled graph including an acceptable goal as a reference Identify the corrective actions

34. Islamabad - 2008 34 Measuring performance Does it help to improve?

35. Islamabad - 2008 35 Impact of public release of performance data Baker Med Care 2002 Analysis of mortality trends during a period (1991–1997) when the Cleveland Health Quality Choice program was operational. Medicare patients hospitalized with 6 medical situations: –acute myocardial infarction (AMI; n = 10,439), –congestive heart failure (CHF; n = 23,505), –gastrointestinal hemorrhage (GIH; n = 11,088), –chronic obstructive pulmonary disease (COPD; n = 8495), –pneumonia (n = 23,719), –stroke (n = 14,293). Measures. –Risk-adjusted in-hospital mortality, –early postdischarge mortality (between discharge and 30 days after admission), –30-day mortality.

36. Islamabad - 2008 36 Results

37. Islamabad - 2008 37

38. Islamabad - 2008 38 Discussion « our findings show that there is still much to learn about what public policies and private initiatives will accelerate improvements in care for medical conditions. »

39. Islamabad - 2008 39 The effect of publicly reporting hospital performance on market share and risk- adjusted mortality at high-mortality hospitals Baker et al, 2003, Med Care.  Despite CHQC's strengths, identifying hospitals with higher than expected mortality did not adversely affect their market share or, with one exception, lead to improved outcomes.  This failure may have resulted from  consumer disinterest  difficulty interpreting CHQC reports,  unwillingness of businesses to create incentives targeted to hospitals' performance,  hospitals' inability to develop effective quality improvement programs.

40. Islamabad - 2008 40 Conclusion Performance indicators should be collected Several indicators should be looked at Measuring performance is a managerial tool

41. Islamabad - 2008 41

42. Islamabad - 2008 42

43. Islamabad - 2008 43 Assessing Performance of ICU A directional distance function approach at the patient level B Dervaux, V Valdmanis, B Guidet What is the benchmark ? What is the productivity of each ICU ? How to deal with variation in case mix ? How to integrate outliers ?  Data envelopment Analysis method : measure of economic efficiency

44. Islamabad - 2008 44 Method Estimation of an efficient frontier that measure technical inefficiency of each patient by the use of relevant directional distance function. An ICU is technically inefficient in treating a patient if it does not minimize its inputs given its outputs. The measure of an ICU’s performance is the sum of its’ patient’s inefficiencies. Chart presenting Econometric performance together with SMR

45. Islamabad - 2008 45 Estimation of a nonparametric production frontier

46. Islamabad - 2008 46 Results Mean inefficiency varies from 19% to 36% The economic inefficiency is concentrated on few patients : –80 % of resources are concentrated in 30% of patients –80 % of inefficiencies are concentrated on less than 20 % of patients. Diagnosis that account for inefficiency : –ARDS, COPD, AIDS, acute renal insufficiency

47. Islamabad - 2008 47 Ressources savings versus SMR SMR Economic inefficiency