1. Differences in adverse events detected using different methods of identification? James M Naessens; Claudia R Campbell; Bjorn Berg; John J Lefante; Arthur R Williams; Richard A Culbertson Division of Health Care Policy & Research, Mayo Clinic, Rochester MN School of Public Health and Tropical Medicine, Tulane University, New Orleans LA

2. Reporting of Patient Safety Measures Hospital acquired conditions, patient safety indicators and “never” events are gaining more attention in public venues. Hospital acquired conditions, patient safety indicators and “never” events are gaining more attention in public venues. Current estimates of medical errors are believed to be substantially undercounted. Current estimates of medical errors are believed to be substantially undercounted. Standard methods and definitions have not always been the norm in reports on patient safety. Standard methods and definitions have not always been the norm in reports on patient safety.

3. Study Objective Determine the degree of congruence between several measures of identifying adverse events among hospitalized patients. Determine the degree of congruence between several measures of identifying adverse events among hospitalized patients.

4. Definitions Medical error – mistake or failure of the care process Medical error – mistake or failure of the care process Adverse event – result of an act of commission or omission with unintended harm to the patient Adverse event – result of an act of commission or omission with unintended harm to the patient Walshe, QSHC (2000) Walshe, QSHC (2000) Negativity Negativity Patient impact Patient impact Causation Causation

5. Measures of Error & Adverse Events 1. Morbidity and mortality conferences 2. Malpractice claims 3. Error reporting 4. Administrative data 5. Medical chart review 6. Electronic medical record 7. Direct observation 8. Clinical surveillance 8. Clinical surveillance Thomas and Petersen, J Gen Int Med, 2003

6. Measures of Error & Adverse Events 1. Morbidity and mortality conferences 2. Malpractice claims 3. Error reporting – provider-reported adverse events 4. Administrative data – AHRQ PSIs 5. Medical chart review – trigger tool 6. Electronic medical record 7. Direct observation 8. Clinical surveillance – bacteremias 8. Clinical surveillance – bacteremias Thomas and Petersen, J Gen Int Med, 2003

7. Methodology Data Sources – Adverse Events All hospital inpatients at Mayo Clinic Rochester hospitals discharged in 2005 with research access authorization (N= 60,599). All hospital inpatients at Mayo Clinic Rochester hospitals discharged in 2005 with research access authorization (N= 60,599). AHRQ PSI – administrative data AHRQ PSI – administrative data Reported event – concurrent staff pager system Reported event – concurrent staff pager system Bacteremia – active surveillance Bacteremia – active surveillance Trigger tool – retrospective review of 10 records every two weeks (N=235) Trigger tool – retrospective review of 10 records every two weeks (N=235)

8. AHRQ Patient Safety Indicators (PSI) Based on computer algorithms applied to secondary diagnosis and procedure codes Based on computer algorithms applied to secondary diagnosis and procedure codes Mixture of hospital complications and preexisting comorbidities Mixture of hospital complications and preexisting comorbidities Incorporated present on admission (POA) indicator Incorporated present on admission (POA) indicator Small difference in definition from presently mandated code since 10/1/2007 Small difference in definition from presently mandated code since 10/1/2007

9. Reported Adverse Events Reporting system – centralized “event pager” carried by RN Reporting system – centralized “event pager” carried by RN Events categorized into medication, equipment, falls, skin events and miscellaneous Events categorized into medication, equipment, falls, skin events and miscellaneous Harm measure recorded at time of capture on all but skin events Harm measure recorded at time of capture on all but skin events Reported “near miss” and events without harm were excluded Reported “near miss” and events without harm were excluded

10. IHI Trigger Tool Adverse Events Nurse review of medical record looking for ~55 “triggers” (e.g., INR > 6) Nurse review of medical record looking for ~55 “triggers” (e.g., INR > 6) If trigger discovered, more intense case review to determine if adverse event related to trigger If trigger discovered, more intense case review to determine if adverse event related to trigger 2 independent nurses reviewed every case and came to consensus 2 independent nurses reviewed every case and came to consensus Cases presented to physician for determination of adverse event and resulting level of harm Cases presented to physician for determination of adverse event and resulting level of harm

11. Results Occurrence of Adverse Events SourceN% Reported Event9221.5 PSI15762.6 Bacteremia800.1 Any event24424.0 Trigger tool6527.7

12. Occurrence of Adverse Events Population: 60,599 Any Event N=2442 PSI Reported Event Blood Stream Infection N=1452 N=817 N=93 6% of PSI N=41 N=8 N=27 N=4 **Diagram is not proportionately correct**

13. Most Frequent Types Reported Events Reported Events Skin N=399* Skin N=399* Medication N=207 Medication N=207 Falls N=190 Falls N=190 Miscellaneous N=342 Miscellaneous N=342 * - All considered, no harm scale PSI PSI Puncture / laceration N= 761 Postop DVT/PE N= 196 Postop Hemorrhage N= 124 Postop Resp Arrest N= 91

14. Skin Events, 2005 PSI 3. Decubitus Ulcer Reported Skin Event Reported by Both Methods N = 52 N = 399 N = 8

15. Decubitus Ulcers 2008 Minnesota Reportable Events Minnesota Reportable Events 3 rd or 4 th degree (and unstageable) skin ulcers : 16 cases 3 rd or 4 th degree (and unstageable) skin ulcers : 16 cases Only 4 have secondary diagnosis code reflecting decubitus ulcer Only 4 have secondary diagnosis code reflecting decubitus ulcer PSI #3 PSI #3 153 cases with secondary diagnosis 153 cases with secondary diagnosis 57 not present on admission, 1 unk. 57 not present on admission, 1 unk.

16. Infections PSI 7. Selected Infections Bacteremias Reported by Both Methods N = 82 N = 80 N = 14 PSI 13. Postop Sepsis Bacteremias Reported by Both Methods N = 48 N = 80 N = 3 14 patients with bacteremia had other PSI 39% of all bacteremias had any PSI

17. Trigger tool cases 65 cases found with adverse event, most frequent cause UTI, majority with temporary harm. 65 cases found with adverse event, most frequent cause UTI, majority with temporary harm. 12 of the 14 reviewed cases with an adverse event found through another method had a trigger-identified event. One PSI and one reported skin event were not detected through trigger tool. 12 of the 14 reviewed cases with an adverse event found through another method had a trigger-identified event. One PSI and one reported skin event were not detected through trigger tool.

18. Limitations Only one location Only one location Inpatient care only Inpatient care only Homogeneous delivery care system with limited number of discharge diagnoses collected (15) Homogeneous delivery care system with limited number of discharge diagnoses collected (15) Limited medical records review due to budget constraints Limited medical records review due to budget constraints

19. Summary Use of patient safety measures in pay for performance, public reporting, other applications need to be aware of limitations of their data collection method. Use of patient safety measures in pay for performance, public reporting, other applications need to be aware of limitations of their data collection method. Adverse events based on diagnosis codes from administrative data differ from cases identified from provider reported systems, even for similar events (infections, falls, decubitus ulcers) Adverse events based on diagnosis codes from administrative data differ from cases identified from provider reported systems, even for similar events (infections, falls, decubitus ulcers)

20. Summary 4% of cases with an adverse event identified through both PSI and reported events (higher when including trigger tool) 4% of cases with an adverse event identified through both PSI and reported events (higher when including trigger tool) - Multiple methods recommended to identify adverse events for internal improvement