International Consensus on Defining and Measuring Quality in Surgical Training Pritam Singh, Rajesh Aggarwal, Boris Zevin, Teodor Grantcharov, Ara Darzi Presented by Pritam Singh

Quality of Surgical Training Background Quality of Surgical Training Quality of Surgeon Patient Outcomes Proposed reforms to educational commissioning in the UK Emphasis on hospitals and surgical training programs to demonstrate evidence of high quality surgical training. Focus has been on quantity

Objectives Improve Surgical Training Define high quality surgical training Define methods to measure quality Measure training quality

Methods Thematic Analysis of 20 semi-Structured Interviews Develop statements Indicators of training quality Methods to measure them Identify international experts in Surgical Education Invitations to online questionnaire Rank each statement Likert scale 1-5 Consensus predefined 80% experts ranking the statement 4 or 5 Cronbach’s alpha 0.80

Results International Consensus 20 Semi-structured interviews 70 statements 83 invitations to experts from 13 countries 47 (57%) responses from 9 countries International Consensus Consensus was achieved on 33 of the 70 statements (Cronbach’s α = 0.90).

Consensus: Focus on Quality Measuring the quality of surgical training will: Have an overall positive effect on training. Encourage competition and drive up standards of training Create accountability for individual trainers. How should the results of measuring surgical training quality be used? Underperforming trusts contacted. Firstly, focus on why measuring training quality is important – and in fact is it important? Secondly, how should we use the results of this exercise? In the current changing climate of educational commisioning, particularly in the UK where trusts will need to demonstrate high quality training in order ot receive trainees, if they underperform, they should be given the information needed to improve. Having established that there is a need and a purpose for measuring the quality of surgical training – the next step is to define high quality surgical training

Consensus: Defining Quality Relationship between the trainer and trainee Good rapport is essential. Trainer must be approachable. Trainer must be willing to allow the trainee to perform operative cases. Exposure to operative cases Case mix. Supervision versus Independence Level of supervision Balance between supervision and independence Facilitates an increasing level of independence for the trainee. Relationship was also the most common theme from the interviews, particulalry when subjects talked of their best training expereinces Case mix is an important marker of surgical training quality. E.G. A good mix of minimally invasive or laparoscopic surgery alongside open surgery. High quality training facilitates an increasing level of independence for the trainee. Supervision may be necessary at the start, but increasing independence can be in the form of supervision from the coffee room Some examples about statements

Consensus: Defining Quality Constructive Feedback Trainers should offer regular, constructive feedback to the trainee. Trainers should be given feedback about their training ability. Strong engagement of the trainer with the training process Structure and Organisation Rota can determine surgical training quality. Challenge the trainee to make progress Personalised training Training placements Trainers should offer regular, constructive feedback to the trainee. This can be in a combination of informal or written feedback Trainers should be given feedback about their training ability. – They need to be offered the opportunity to improve Explain rota statement – for instance, shift patterns can affect the continuity between the trainee and trainer and subsequently affect the rapport A high quality surgical training post should challenge the trainee to make progress during the post

Consensus: Defining Quality Simulation and Teaching Structured teaching programme Teamwork Trainee rapport with other trainees and surgeons Stepwise approach to training Clear objectives set jointly by the trainer and trainee Operating Room Lists structured to take into account time for trainees. Involved in all operative cases even if complicated. Repetition of similar procedures. It is essential for training placements to be matched to the individual trainee's level. Presence of a structured teaching programme is a marker of surgical training quality. The role of simulation within this teaching programme was less well defined. Trainee rapport with other trainees and surgeons affects the quality of surgical training. E.g. Senior trainees teaching/mentoring junior trainees A high quality surgical training post starts with clear objectives set jointly by the trainer and trainee Reference the costs of OR training time which were discussed earlier today

Consensus: Measuring Quality Trainee feedback Aggregate measures Formative evaluations of their trainers. Multi-source feedback? Trainee feedback. Trainer feedback. Trainee improvement Evidence during each training post. Formative evaluations of trainee from trainers Practical technical assessment. Include assessment of non-technical skills. Aggregate measures from a number of trainees should be used. Avoids indivdual personality clashes Trainees should provide formative evaluations of their trainers. Multi-source feedback to measure surgical training quality? Trainee feedback is an essential component. Trainer feedback is an essential component. – they often know the limitations of their training program and would welcome the opportunity to try and improve them. There should be actual evidence of trainee improvement during each training post. This includes formative evaluations of trainee from trainers Formative assessment should include practical technical assessment of trainees. Formative assessment should include assessment of non-technical skills of trainees. Written assessments were described as being difficult to pick up these factors

Conclusion International expert consensus to define Surgeons believe Elements of high quality surgical training Metrics for their measurement. Surgeons believe Comparisons of Surgical Training Quality Create Competition Drive up Standards of Surgical Education

Future Research Improve Surgical Training Define high quality surgical training Define methods to measure quality Measure training quality Develop a tool to objectively evaluate quality of surgical training programs

Thank You