An Objective Measurement of Training ? Quality Panels: An Objective Measurement of Training ? Jon Francis Associate Postgraduate Dean Quality HEE Training The New Generation 18th January 2017

Session Objectives Learn about a way of assessing the quality of PGME Understand how to run a Quality Panel Evaluate the Merits of Quality Panel Assessment Consider the future role of Quality Panels

Quality Panel: Why ?

Quality Panel: Why ? GMC survey imperfect tool Trainee voice is heard Triangulation with other sources of quality data

Quality Panel History How to run a quality panel HEE South West Experience Quality Panel future

Who attends the panel meeting? Training programme director or speciality tutor Trainee Educational supervisor Lay representative Quality Panel The meeting was chaired by the TPD or speciality tutor. There was a lay rep, educational supervisors and trainees. A key point was the requirement to have more trainees than trainers on the panel.

What happened at the panel meeting? Review of data from returned trainee matrices and overall gradings allocated by reps Review of quality grading and associated information Discussion around overall grading to be awarded Discussion around areas of excellence and improvement Discussion around how to take the post forward The panels were structured in slightly different ways according to the set up of the training programme being evaluated. Generally speciality schools were structured in similar ways, GP slightly different and foundation different again due to the nature of the individual training programmes. In brief, all trainees on a given programme completed two evaluation matrices which were based on the new GMC guidance: Promoting Excellence, standards for medical education and training. The matrices involved trainees grading their training post as excellent, good requires improvement or inadequate for various domains. (cut to next slide) with the option of comments. The gradings and comments from these matrices were returned to the school teams and collated into an excel spreadsheet.

Effective Educational Environment

Safe Supportive Working Environment

Example 1

Example 2

Example 3

Effective Education Grade Overall Quality Grade Safe Working Grade Quality Data Effective Education Grade

Quality Panel BOS experience

Overall Quality Grade POST Safe Supportive Working Environment Supportive Educational Environment External Quality Data Overall   Foundation 1 Gastro

Quality Panel Review Anomaly Grade What about stand out issues not consistent with overall grade? Capture outlining issues requires action and recording

Quality Panel Review: Reports Standardised report template for review Recording of grading, comments, actions Anomalies Report written as a ‘public document’ Emphasis on actions need to improve scoring, whatever the score. Raising the ceiling as well as the floor SMART objectives

Quality Panel Review: Pros Useful opportunity to review ‘non GMC survey’ training data Useful opportunity to triangulate quality data A powerful process for improving quality of training Opportunity for greater trainee involvement

Quality Panel Review: Cons a lot of work Consistency across all schools Survey fatigue trainees Future conflict with HEE NETS survey ?

Session Objectives Learn about a way of assessing the quality of PGME Understand how to run a Quality Panel Evaluate the Merits of Quality Panel Assessment Consider the future role of Quality Panels

Summary and conclusions Quality Panels are an effective and helpful way of measuring the Quality of PGME A powerful tool in driving up the quality of PGME Enables trainee engagement in their programme Work load, could it replace other activities ? Future ?