Assessment of Team Skills in the Operating Room: Development and Evaluation of a ‘Train- the-Trainers’ Program Dr Louise Hull Research Associate, Department of Surgery and Cancer, Imperial College London

Non-Technical Skills, Teamwork and Performance

ASSESSING NTS & TEAMWORK IN THE OR Availability of assessment tools Availability of faculty TRAINED to assess Assessment

Availability of Assessment Tools OTAS NOTSS, __ ANTS, __SPLINTS Revised NOTECHS Oxford NOTECHS ASSESSING NON-TECHNICAL & TEAM SKILLS IS A SKILL IN ITSELF

Training Faculty to Assess and Debrief is CRITICAL Learning from Other Industries LACK OF GUIDELINES/RECOMMENDATIONS CURRENTLY EXIST FOR TRAINING FACULTY TO ASSESS AND DEBRIEF IN SURGERY

6 Training Guidelines For The OR AIMS: 1.To obtain expert consensus on the specifications of a ‘train-the-trainers’ program for Faculty assessing non- technical and team skills 2.To develop, pilot and evaluate a ‘train-the-trainers’ program (based on expert guidelines & recommendations)

Methodology UK-wide multistage expert consensus study Experts in non-technical/team skills assessment (n=19) 11 clinicians (8 surgeons, 3 anaesthesiologists) and 8 psychologists/human factors specialists 3 Stage Delphi Stage 1: Interviews Interviews to explore expert views on Training Requirements Stage 2: Survey (Round 1) Analysed for agreement and consensus Stage 3: Survey (Round 2) Survey repeated, identical to first (but incorporating group's second round responses

Training Program Development Based on Expert Guidelines/ Recommendations

Results: Training Guidelines Trainers: Who should deliver the ‘Train-the-Trainers’ program? Multi-disciplinary team consisting of clinicians and psychologists/human factors experts Faculty: Who should the ‘Train-the-Trainers’ programme be aimed at? Senior clinicians (attending-level) Clinicians that regularly provide training and assessment Clinicians and psychologists/human factors experts (alongside clinicians)

Training Guidelines Theoretical background on non-technical skills and human factors applied to healthcare/surgery Introduction to non-technical skill assessment tools Limitations of non-technical and team skill assessment (e.g. rating errors) Implications of non-technical and team skill assessments Training in the recognition of non-technical and team skills Practice in rating non-technical and team skills Training in providing feedback/debriefing following a non-technical and team skill assessment Training Program Content

Training Guidelines Proficiency To provide performance feedback within clinical practice: Minimum intra-class correlation coefficients of To provide high-stakes assessments: Minimum intra- class correlation coefficients Recertification Faculty should be assessed annually if providing high- stakes assessments Methods: Video clips for practising observing and rating non-technical skills Duration: 2 full-days, proficiency-based training, support after initial training, and refresher course(s) Ratio of Trainers to Faculty: 1:12 or more for theoretical components, 1:4-6 for practical components of the training. Training Programme Delivery

Training Program Development Based on Expert Guidelines/ Recommendations Trainers: Multidisciplinary (Clinicians & Psychologists) Faculty: Senior Clinicians (surgeons, anaesthesiologists, OR nurses) Duration: 1 day Delivery: Workshop Content: Theoretical and practical components Team Assessment: Observational Teamwork Assessment for Surgery (OTAS)

Observe Teamwork Video Rate Teamwork Behaviours using OTAS Group Discussion followed by Presentation of Expert Ratings Practice Observing and Rating Team Skills

28 faculty attended two training sessions (16 and 12 respectively) Mix of surgeons, anaesthesiologists, nurses, and OR managers Attending & Senior Residents Faculty

Discussion & Conclusion First training program, developed according to national consensus guidelines, which suggest that faculty are able to reach an acceptable level of proficiency in teamwork assessment Assessing non-technical skills/performance is a skill in itself – and thus it requires training Training faculty will be essential to formally integrate team skills assessment into surgical programs ASSESSING TEAMWORK IN THE OR Availability of assessment tools Availability of faculty TRAINED to assess

Expert Consensus Panel Sonal Arora Andrew Healey Eva Kassab Krishna Moorthy Nick Sevdalis Shabnam Undre Professor Charles Vincent Associate Professor Ken Catchpole Peter McCulloch Amitabh Mishra Ronnie Glavin Professor Rhona Flin Nikki Maran Lucy Mitchell Simon Paterson-Brown Rona Patey Professor David Rowley Steve Yule Professor George Youngson Research Team and Training Faculty Sonal Arora Professor Bryn Baxendale Professor the Lord Ara Darzi Rohz Jalil Peter McCulloch Krishna Moorthy Stephanie Russ Nick Sevdalis Nick Symons Professor Charles Vincent Acknowledgements

Thank you for your attention!