Interrater agreement and time it takes to assign a Canadian Triage and Acuity Scale score in 7 emergency departments McLeod, S.L.1,2 McCarron, J.3 Stein, K.3 Al-Mujtaba, M.3 Scott, S.3 Ovens, H.1,2 Mittmann, N.3 Borgundvaag, B.1,2 1) Schwartz/Reisman Emergency Medicine Institute, Mount Sinai Hospital 2) Department of Family and Community Medicine, University Of Toronto 3) Analytics & Informatics, Cancer Care Ontario Reliability Study Purpose Preliminary Results Figure 1: Generating a CTAS Score The Canadian Triage and Acuity Scale (CTAS) is used in all Canadian emergency departments (EDs) for establishing the priority by which patients should be assessed In 2010, 38% of patient cases in Ontario were under-triaged1 In 2015, the Ministry of Health and Long-Term Care engaged CCO to develop and implement a provincial electronic Canadian Triage Acuity Scale (eCTAS) system to improve consistency in CTAS application In 2016, in preparation of the implementation of eCTAS, CCO conducted two evaluation studies to establish baseline data regarding 1) reliability of CTAS assessments, and 2) triage times at representative hospitals using both single and double stage triage Reliability Study Results (Table 1) indicates: 738 consecutive patient CTAS assessments were audited Exact modal agreement was achieved for 554 (75.0%) patients On-duty triage nurses: Over-triaged 89 (12.1%) patients Under-triaged 95 (12.9%) patients Interrater agreement was “good” Unweighted kappa of 0.63 (95% CI: 0.58, 0.67) Quadratic-weighted kappa of 0.79 (95% CI: 0.67, 0.90) Time Study Results (Table 2) indicates: Overall median (IQR) triage time was 5.2 (3.8, 7.3) minutes Triage time varied from 3.9 (3.1, 4.8) minutes to 7.5 (5.8, 10.8) minutes Median (IQR) time for: Single-stage triage: 4.4 (3.5, 6.0) minutes Two-stage triage: 7.3 (5.6, 10.0) minutes Table 1: Interrater Agreement Table 1: Interrater Agreement Expert CTAS Auditor 1 2 3 4 5 Total Triage Nurse 121 27 151 247 43 318 55 168 13 238 7 18 154 331 220 31 738 Methods Study Design: Observational studies in 7 hospital EDs, pre-post eCTAS implementation. Pre-Implementation Reliability Study (Fig. 1): Study periods covered three 7-hour shifts per hospital (21 shifts total) Triage nurse and expert auditor blinded to each other’s CTAS decisions, co-triaged the same patients and assigned CTAS scores Expert auditor documented: Presenting complaint captured by the triage nurse Canadian Emergency Department Information Systems (CEDIS) presenting complaint recommended by the expert auditor Vital signs captured at triage with patient age CTAS level determined by the duty triage nurse Modifiers observed by the expert auditor CTAS level determined by the expert auditor Interrater agreement was estimated using unweighted and quadratic-weighted kappa statistics with 95% confidence intervals (CIs). Time Study (Fig. 2): Study periods covered approx. ten 6-hour shifts per hospital (69 shifts total) Triage time elapsed from first patient-nurse interaction to the time the triage assessment is complete Triage timestamps were captured by research students Post-Implementation For both studies: Observations in each hospital ED will be conducted at a minimum of three months after eCTAS implementation (May to October 2017) Each site will serve as it's own control when pre-post implementation data will be compared Conclusion Variability in the accuracy and time taken to perform CTAS assessments suggest that a standardized approach to performing the CTAS process would improve both clinical decision making and administrative data accuracy. Further Analyses: The eventual comparison of pre-post eCTAS implementation data will provide evidence on how the eCTAS process could significantly improve CTAS reliability without increasing triage time. Yellow – under triage, Green – Agreement, Red – over triage Time Study Figure 2: Triage Time Evaluation Setting References 1. Office of the Auditor General of Ontario, 2010. 2010 Annual Report (3.05): 132-163. http://www.auditor.on.ca/en/content/annualreports/arreports/en10/305en10.pdf Table 2: Median Triage Time by Hospital Hospital #Visits #Patients Median (IQR) Triage Time (min) A 10 (14.5%) 440 (11.6%) 5.5 (4.0, 7.4) B 582 (15.3) 4.3 (3.5, 5.7) C 9 (13.0%) 472 (12.4%) 7.5 (5.8,10.8) D 300 (7.8%) 5.9 (4.5, 7.7) E 384 (10.1%) 4.4 (3.5, 5.7) F 804 (21.1%) 3.9 (3.1, 4.8) G 826 (21.7%) 6.9 (5.2, 9.2) Total 69 3808 5.2 (3.8, 7.3) Acknowledgement This work is supported by the Ministry of Health and Long-Term Care.