Agreement among healthcare professionals in diagnosing case vignette-based surgical site infections in 10 European countries. G. Birgand 1, D. Lepelletier 2, G. Baron 3, S. Barrett 4, A-C Breier 5, C. Buke 6, L. Markovic-Denic 7, P. Gastmeier 5, J. Kluytmans 8, O. Lyytikainen 9, E. Sheridan 10, E. Szilagyi 11, E. Tacconelli 12, N. Troillet 13, P. Ravaud 3, JC Lucet 1 and the VARSSI study group 1 Infection control unit, Bichat-Claude Bernard Hospital, Paris, France; 2 Hygiene Department, Nantes University Hospital, France; 3 Centre d’Épidémiologie Clinique, Hôpital Hôtel Dieu, Paris, France; 4 Medical microbiology and infection control, Southend Hospital, UK; 5 Institute of Hygiene and Environmental Medicine, Charité - University Medicine Berlin, Germany; 6 Department of Infectious Diseases and Clinical Microbiology, Ege University Medical Faculty, Izmir, Turkey; 7 Institute of epidemiology, Faculty of Medicine, University of Belgrade, Serbia; 8 Medical microbiology and infection control, Amphia medical center Breda, The Netherlands; 9 Department of Infectious Disease Surveillance and Control, National Institute for Health and Welfare (THL), Helsinki, Finland; 10 Department of Healthcare-Associated Infection and Antimicrobial Resistance, HPA Centre for Infections, London, UK; 11 Department of Epidemiology, Budapest, Hungary; 12 Department of Infectious Diseases, Università Cattolica del Sacro Cuore, Rome, Italy; 13 Department of infectious diseases, Central Institute, Valais Hospital, Sion, Switzerland.

Disclosure statement Financial support: – French Ministry of Health (national grant PREQHOS 0901) Conflict of interest: – Pfizer: Travel grant for the ICAAC 2011 G Birgand VARSSI study ECCMID Berlin 2013

Background Public reporting of HAI to improve public information and performance: process indicators or infection rates However, SSI rates vary according to several factors – Co-morbidities, contamination class, conditions of the surgical procedure – Robustness of SSI rates depends on definitions, training, and experience Recent studies: – Wilson AP, BMJ 2004: variable interpretations of definitions – French study: disagreement among healthcare professionals in SSI diagnosis using case-vignettes (Lepelletier, Plos one 2012) – European network: large differences in SSI recognition across countries (Wilson J, JHI 2007) G Birgand VARSSI study

Objectives To assess agreement in SSI diagnosis among Infection Control Physicians (ICPs) and surgeons involved in SSI surveillance in 10 European countries - Main objective = intra-country/intra-specialty variability - Secondary objective = intra-specialty/inter-country variability - Criteria n°1: presence/absence of SSI - Criteria n°2: depth To assess the impact of SSI definition availability on diagnosis G Birgand VARSSI study ECCMID Berlin 2013

10 participating countries G Birgand VARSSI study

Methods Elaboration of case vignettes 20 case vignettes built from patients hospitalised in 6 surgical units of 3 French university hospitals – Cardiac (n=5), Digestive (n=5), Orthopedic (n=4), ENT (n=2), Gynecology (n=2) and Neurosurgery (n=2) – Wound discharge and/or infectious signs – Bedside evaluation with collection of biological, bacteriological and radiological signs Case vignettes standardised by the same investigator – Same chronological information – Internet-secured relational database with personal login and password ECCMID Berlin 2013 G Birgand VARSSI study

20 vignettes 10 ICPs x 8 vignettes 10 Surgeons x 8 vignettes 20 x 4 = 80 readings per specialty 160 readings per country X10 countries 1600 readings

No SSISSI SSI certainly absent SSI certainly present Existence of SSI Intra-class correlation coefficient (ICC) with 95%CI Methods G Birgand VARSSI study ECCMID Berlin 2013

No SSISSI SSI certainly absent SSI certainly present Superficial SSI Deep or organ/space SSI Don’t know Existence of SSI Intra-class correlation coefficient (ICC) with 95%CI Depth of SSI kappa coefficient (kC) with 95%CI Methods G Birgand VARSSI study

Methods Interpretation of agreement Intra-correlation coefficient (presence/absence of SSI) – < 0.40= poor agreement – = good agreement – > 0.71= excellent agreement Kappa coefficient (SSI depth) – < 0.40= poor agreement – = fair agreement – = good agreement – > 0.81= excellent agreement Landis JR. Biometrics 1977 Nunnally JC. Psychometric theory 1994 G Birgand VARSSI study ECCMID Berlin 2013

Number of participants 10 ICPs + 10 Surgeons 10 ICPs + 4 Surgeons 10 ICPs + 9 Surgeons 100 ICPs and 86 Surgeons G Birgand VARSSI study

Results Assessment of agreement about SSI diagnosis G Birgand VARSSI study Intra-country correlation N SSI diagnosis scoring, 7-point Likert scale Intra-correlation coefficient ICC ICP Turkey Serbia Hungary Italy Germany Finland Netherlands Switzerland France UK Inter-country

G Birgand VARSSI study Results Assessment of agreement about SSI diagnosis Intra-country correlation N SSI diagnosis scoring, 7-point Likert scale Intra-correlation coefficient ICC ICPSurgeons Turkey Serbia Hungary Italy Germany Finland Netherlands Switzerland France UK Inter-country

Intra-country correlation N SSI diagnosis scoring, 7-point Likert scale Intra-correlation coefficient ICC ICPSurgeonsAll Turkey Serbia Hungary Italy Germany Finland Netherlands Switzerland France UK Inter-country G Birgand VARSSI study Results Assessment of agreement about SSI diagnosis No impact of definition on the agreement between participants regarding the presence of SSI

Results Assessment of agreement about SSI depth G Birgand VARSSI study Countries SSI depth, 4-point scale (Kappa coefficient) ICP Turkey 0.26 Serbia 0.05 Hungary 0.20 Italy 0.23 Germany 0.50 Finland 0.27 Netherlands 0.25 Switzerland 0.34 France 0.44 UK 0.38 Inter-country 0.28

Results Assessment of agreement about SSI depth G Birgand VARSSI study Countries SSI depth, 4-point scale (Kappa coefficient) ICPSurgeons Turkey Serbia 0.05 Hungary Italy Germany Finland Netherlands Switzerland France UK Inter-country

Countries SSI depth, 4-point scale (Kappa coefficient) ICPSurgeonsAll Turkey Serbia Hungary Italy Germany Finland Netherlands Switzerland France UK Inter-country Results Assessment of agreement about SSI depth G Birgand VARSSI study No impact of definition on the agreement between participants regarding SSI depth

Discussion Large disagreement among healthcare professionals in SSI diagnosis: – varied both between and within specialties  need for a multidisciplinary approach One country (Germany) with the best agreement – Regular cross-hospital evaluation of diagnosis accuracy through surveillance network Limitations: – SSI diagnosis made by individuals without teamwork – Scoring via an online database without contact with patients – Selection bias (recruitment by European leaders of participants presumed to be expert on this topic) G Birgand VARSSI study

Conclusion Original approach to assess the variability of SSI diagnosis – Large panel of ICPs and surgeons in 10 European countries SSI rates should be used with caution for benchmarking or public reporting SSI surveillance and feedback remain important tools for SSI prevention – Further studies are needed to improve agreement regarding the SSI diagnosis G Birgand VARSSI study ECCMID Berlin 2013

Aknowledgment The VARSSI study group: –Turkish, Finnish, Hungarian, German, Italian, Serbian, Swiss, Duch, British and French ICPs and Surgeons participants Participating wards: –Pr Nataf, Cardiac surgery, Pr Marmuse, Digestive surgery; Pr Massin, Orthopedic surgery, Bichat- Claude Bernard University Hospital –Pr Despins, Cardiac Surgery, University Hospital, Nantes; G Birgand VARSSI study ECCMID Berlin 2013

Thank you for your attention

Results Distribution of scores CountriesN Infect Contr PhysSurgeons Median (IQR) Turkey (4 - 7)7 (4 - 7) Serbia (3 - 7) 6 (3 - 7) Hungary (4 - 7) 7 (3 - 7) Italy (4 - 7) 6 (3 - 7) Germany ( ) 6 (3 - 7) Finland (3 - 7) Netherlands (3 - 7) 6 (4 - 7) Switzerland ( ) 7 (6 - 7) France (3 - 7) 5 (2 - 7) UK (4 - 7) 6 ( ) Total (4 - 7) 6 (3 - 7) G Birgand VARSSI study

Results Distribution of scores CountriesN Infect Contr PhysSurgeons Median (IQR)Min-MaxMedian (IQR)Min-Max Turkey ·0 (4·0-7·0)1·0-7·07·0 (4·0-7·0)1·0-7·0 Serbia ·0 (3·0-7·0)1·0-7·0 6·0 (3·0-7·0)1·0-7·0 Hungary ·0 (4·0-7·0)1·0-7·0 7·0 (3·0-7·0)1·0-7·0 Italy ·0 (4·0-7·0)1·0-7·0 6·0 (3·0-7·0)1·0-7·0 Germany ·0 (3·5-7·0)1·0-7·0 6·0 (3·0-7·0)1·0-7·0 Finland ·0 (3·0-7·0)1·0-7·0 6·0 (3·0-7·0)1·0-7·0 Netherlands ·0 (3·0-7·0)1·0-7·0 6·0 (4·0-7·0)2·0-7·0 Switzerland ·0 (4·5-7·0)1·0-7·0 7·0 (6·0-7·0)1·0-7·0 France ( )1·0-7·0 5.0 ( )1·0-7·0 UK ·0 (4·0-7·0)1·0-7·0 6·0 (4·5-7·0)1·0-7·0 Total1488 6·0 (4·0-7·0)1·0-7·0 6·0 (3·0-7·0)1·0-7·0 G Birgand VARSSI study

Intra-country correlation N SSI diagnosis scoring, 7-point Likert scale Intra-correlation coefficient ICC ICP Spontaneous scoring Surgeons Spontaneous scoring All Spontaneous scoring Turkey ·44 (0·22-0·68) 0·23 (0·03-0·50) Serbia ·26 (0·05-0·53)0·06 (0·00-0·35) 0·20 (0·04-0·41) Hungary ·28 (0·07-0·54)0·24 (0·04-0·51)0·31 (0·16-0·54) Italy ·31 (0·10-0·57)0·08 (0·00-0·36) 0·20 (0·07-0·42) Germany ·65 (0·45-0·82)0·46 (0·23-0·69)0·55 (0·37-0·74) Finland ·30 (0·09-0·57)0·21 (0·01-0·48) 0·30 (0·15-0·52) Netherlands ·45 (0·23-0·69)0·00 (0·00-0·35)0·42 (0·23-0·65) Switzerland ·40 (0·18-0·65)0·08 (0·00-0·34)0·26 (0·12-0·48) France ·44 (0·22-0·68)0·30 (0·07-0·57) 0·41 (0·24-0·63) UK ·59 (0·38-0·80)0·04 (0·00-0·62) Inter-country1488 0·41 (0·28-0·61)0·24 (0·14-0·42) G Birgand VARSSI study Results Assessment of agreement about SSI diagnosis

Countries SSI depth, 4-point scale (Kappa coefficient) Not scored ICP Not scored Surgeons Not scored All Turkey 190·26 (0·21-0·31) 170·25 (0·20-0·30) 360·27 (0·24-0·30) Serbia 210·05 (0·00-0·10)250·05 (0·00-0·10)460·09 (0·07-0·11) Hungary 180·20 (0·15-0·25)220·16 (0·10-0·22)400·19 (0·16-0·22) Italy 210·23 (0·17-0·29)230·09 (0·01-0·17)440·31 (0·18-0·34) Germany 200·50 (0·45-0·55)210·31 (0·26-0·36)410·35 (0·32-0·38) Finland 240·27 (0·22-0·32)230·14 (0·09-0·19)470·25 (0·22-0·28) Netherlands 230·25 (0·20-0·30)70·17 (0·00-0·34)300·25 (0·18-0·32) Switzerland 110·34 (0·28-0·40)180·14 (0·08-0·20)290·24 (0·21-0·27) France 210·44 (0·39-0·49)270·17 (0·09-0·25)480·27 (0·23-0·31) UK 200·38 (0·32-0·44)60·19 (-0·02-0·40)260·34 (0·25-0·42) Inter-country 1980·28 (0·27-0·29)1890·19 (0·17-0·21)-- Results Assessment of agreement about SSI depth G Birgand VARSSI study