Steven Yule, Rhona Flin, George Youngson University of Aberdeen Simon Paterson-Brown, Nikki Maran Royal Infirmary of Edinburgh David Rowley University of Dundee Surgeons non-technical skills: Reliability testing of the NOTSS system Project sponsors: Royal College of Surgeons of Edinburgh (RCSEd) NHS Education for Scotland (NES) Association for Surgical Education, Washington DC, April 2007

Right hepatic artery Common bile duct right hepatic duct Portal vein

“High technical proficiency cannot guarantee safety” Surgeons’ non-technical skills defined: “Behavioural aspects of performance in the operating theatre which underpin medical expertise, use of equipment and drugs. They are cognitive (i.e. situation awareness) and social (i.e. communication & teamwork) skills”

Research on non-technical skills Adverse events in surgery are primarily caused by failures in perception, judgement, communication and teamwork »Wilson et al (1999) - communication breakdown in 43% of errors »de Leval et al (2000) – cardiac switch operations »Way et al (2003) – 97% of bile duct injuries had perception failures »Gawande et al (2003; 2004) – insurance claims in the US Good non-technical skills can lead to positive outcomes for the team and patient »Edmondson (2003) – effective leadership »Carthey et al (2003) – behavioural markers in cardiac surgery »Healey et al (2004) – observing teamwork in surgery »Moorthy et al (2005) – team skills in operating theatre simulator

NOTSS project background Rationale »Support safe practice in the operating theatre and surgeons’ professional training Aim »Identify the essential non-technical skills for surgeons »Develop system for surgeons to rate skills in operating theatre and give structured feedback for training »Establish reliability & usability of system Scope »Develop an education tool »Intraoperative environment only »For surgeons to observe, rate, and provide feedback

Developing the NOTSS system (based on Gordon, 1993, p.12) Phase 1: Task analysis 1. Literature review (Surgery, 2006) 2. Adverse event analyses 3. Cognitive Interviews (n=27 surgeons, n=12 nurses) 4. Observation in theatre 5. Attitude survey (The Surgeon, 2006) Phase 2: Design and development (Medical Education, 2006) Develop skills taxonomy Refine categories and elements (4 panels of surgeons) Write behavioural markers (n=16 surgeons) Prototyping and usability testing Phase 3: System evaluation Finalize evaluation criteria – reliability and usability Design evaluation strategy: rwg, in-theatre trials Collect and analyse data (44 surgeons; c. 100 cases) Modify system if necessary

CategoriesElements Situation Awareness Gathering Information Understanding Information Projecting and anticipating future state Decision MakingConsidering options Selecting and communicating option Implementing and reviewing decisions Communication and Teamwork Exchanging information Establishing a shared understanding Co-ordinating team LeadershipSetting and maintaining standards Coping with pressure Supporting others NOTSS skills taxonomy v1.2 (Yule et al, Medical Education, 2006)

NOTSS System Structure Situation Awareness Decision Making Communication and Teamwork Leadership Gathering Information Understanding Information Projecting and anticipating future state Surgeons’ Non-Technical Skills Skill Categories Skill Elements Behavioural Markers Good: Keeps anaesthetist informed about procedure (e.g. to expect bleeding) Poor: Waits for a predicted problem to arise before responding

NOTSS rating scale 1 Poor Performance endangered or potentially endangered patient safety, serious remediation is required 2 Marginal Performance indicated cause for concern, considerable improvement is needed 3 Acceptable Performance was of a satisfactory standard but could be improved 4 Good Performance was of a consistently high standard, enhancing patient safety; it could be used as a positive example for others N/A Not Applicable. Skill was not required or not relevant in this case or scenario

NOTSS reliability study Purpose »Test how reliably consultant surgeons use NOTSS to rate behaviours in standardized scenarios Scenario development »Good & poor behavioural examples in 9 scenarios »‘case mix’ of orthopaedic & general surgical scenarios »Simulator used as patient, surgeons and nurses as actors Participants »n=44 consultant surgeons (mean experience 8.9 yrs (sd 7.5)) »2-3 hour training session Analysis »Within group agreement (r wg ) »Intraclass correlations (ICC) »Reference/ expert ratings

NOTSS video scenarios Leadership Situation Awareness Communication & Teamwork Decision Making

Reliability study - results Categories*r wg ICC(2)SICC(2)A Elementsr wg Situation Awareness Gathering Information Understanding Information Projecting and anticipating future state Decision Making Considering options Selecting and communicating option Implementing and reviewing decisions Communication and Teamwork Exchanging information Establishing a shared understanding Co-ordinating team Leadership Setting and maintaining standards Coping with pressure Supporting others *rated separately, not an average of element ratings

Critical level

Limitations Novel task for surgeons Limited training for participants Calibration would improve reliability Inter-rater agreement will improve with experience of using the system and non-technical language

Current research Controlled usability trial in the OR, using NOTSS to observe, rate and debrief on performance Examine the relationship between non-technical and technical skills Contact: Project website: