1. PMETB/COPMeD National Survey of Trainees PMETB National Survey of Trainers, 2007.

2. PMETB and COPMED National Trainee Survey 2007 Responses from F2 Doctors Heather Payne¹, Daniel Smith², Elisabeth Paice 3 ¹Wales School of Postgraduate Medical and Dental Education, Cardiff University ²Postgraduate Medical Education and Training Board 3 London Postgraduate Deanery, Chair of COPMED

3. National Trainee Survey 2007- Aims of workshop To share collated feedback from Foundation Trainees in post in 2007 To identify notable practice, and potential barriers to achieving improvement To explore expectations and implications for local practice To explore use of the PMETB Trainee Survey web based reporting tool to run local queries

4. Second annual National Survey of Trainees First ever National Survey of Trainers Methodology –Item development –Consultation Survey launch Response rates Analysis –Launch of the Reporting Tool –National report Introduction

5. PMETB Training standards

6. Aim: To assess compliance against PMETB’s Generic Standards for Training, including the new Standards for Trainers The National Surveys are one part of PMETB’s quality assurance framework designed to enable the board to meet its statutory requirement to maintain and develop the standard of postgraduate medical education and therefore it must lead to quality improvement activity. Background

7. PMETB QF 5. RESPONSES TO CONCERNS Range to ensure patient and/or trainee safety 4. VISITS to deaneries 3. SURVEYS Trainees and trainers 1. STANDARDS Includes approval and action planning 2. SHARED EVIDENCE BASE includes minimum data set

8. Item development Last year’s generic items, plus: –Foundation items –Psychometric items –Specialty specific items Specialty specific items aimed to measure adequate experience. Approximately 5 items approved by the SAC chair or equivalent person. Items consulted on. Methodology: National Survey of Trainees

9. Literature search to obtain relevant items already in use, presented to Trainer Survey Working Group for selection Consultation conducted of chosen items (deaneries and colleges/faculties) Face validity of items conducted by researchers at NES and the Northern Deanery Final amendments and consultation with working group members and deanery contacts. Methodology: National Survey of Trainers Item development

10. ROCR-lite (sic) approval was awarded for both surveys. The surveys have been developed in consultation with The Information Centre for Health and Social Care who considers the data collection to be useful and reasonable. What is ROCR-lite? ROCR-Lite is a service for Concordat signatories and associated bodies. The process is designed to: allow flexibility in meeting the needs of the various organisations it seeks to support to ensure compliance will differ between organisations. ROCR-lite approval – England only

11. Trainee Survey - Pilot of administration process in Wales deanery 15 Nov start; all other deaneries started 4 Dec. The survey closed on 28 Feb to allow time to chase non- respondents and missing deanery data. This year the survey was mandatory for all run-through trainees, as stated in the Gold Guide. Trainer Survey - Pilot of the administration in London 14 Dec, all other deaneries 3 Jan. Survey closed 28 Feb. Both surveys received considerable support from deanery staff and MEM, who kindly provide the necessary data to administer them and promoted them to the respondents. Survey launch

12. Trainee Survey – Emails collected from deaneries together with post data. Emails sent via the database to trainees and reminders sent to non-respondents. Trainer Survey – Deaneries provided a nominated contact such as the DME or equivalent within the LEP. Nominated contacts were then asked to cascade the survey invitation to all consultants at their location. Active GP trainers and Foundation tutors received their invitation directly from PMETB. Survey launch process

13. Why? To assess satisfaction with training Quality assurance using PMETB standards How? Survey run Dec 2007- Feb 2008 All Doctors in UK training posts on 30 Nov 2007 PMETB Trainee survey

14. Survey response rates 2007 Overall response rate (all trainees) 66% (n=33,329) Response rate by Deanery 45-82% Foundation response rate 56% (n=6,897) F2 responses n=3,776

15. Demographic of F2 Trainees 2007 58% female 91% age 30 or under 82% UK graduates 5% intended to leave UK once training complete

16. At start of post, F2 Trainees….. Told about roles and responsibilities Given all the information they need Identified Educational Superviser Discussed educational objectives with Superviser % % % %

17. At start of post, F2 Trainees….. Told about roles and responsibilities Given all the information they need Identified Educational Superviser Discussed educational objectives with Superviser 85% 71% 99% 82%

18. F2 Trainees - Induction Notable practice Barriers to good enough practice What needs to change?

19. F2 Trainees - Education and Assessment Using an Educational Portfolio Confident that post would deliver required competences Experience good or excellent %

20. F2 Trainees - Education and Assessment Using an Educational Portfolio Confident that post would deliver required competences Experience good or excellent 90% 75% 66%

21. F2 Trainees - Education and Assessment Notable practice Barriers to good enough practice What needs to change?

22. Quality of Supervision items How often have you felt forced to cope with problems beyond your competence or experience? How often, if ever, have you been supervised by someone who you feel isn’t competent to do so? How often have you been expected to obtain consent for procedures which you do not carry out yourself? Do you always know who is providing your clinical supervision when you are working? Please indicate your perception of the way in which critical incidents and near misses are reported in your department

23. F2 Trainees - Supervision and Support Formal meeting with superviser on performance in post Receiving informal feedback from senior at least weekly Formal assessment of performance in workplace %

24. F2 Trainees - Supervision and Support Formal meeting with superviser on performance in post Receiving informal feedback from senior at least weekly Formal assessment of performance in workplace 50% 34% 48%

25. F2 Trainees - Supervision and Support Notable practice Barriers to good enough practice What needs to change?

26. F2 Trainees - Career influences Discussed career plans with senior Decided on career intention Would recommend post to a friend Feel post not useful in career %

27. F2 Trainees - Career influences Discussed career plans with senior Decided on career intention Would recommend post to a friend Feel post not useful in career 71% 91% 67% 10%

28. F2 Trainees - Career influences Notable practice Barriers to good enough practice What needs to change?

29. F2 Trainees - Safe Doctors? Critical event/ near miss reporting encouraged and followed up Coping beyond competence weekly or daily % %

30. F2 Trainees - Safe Doctors? Critical event/ near miss reporting encouraged and followed up Coping beyond competence weekly or daily 73% 32%

31. 7. In the last month, have you made a serious medical error? No / Yes, once / Yes, more than once /Do not wish to answer 8. In the last month, have you made a potentially serious medical error? No / Yes, once / Yes, more than once /Do not wish to answer Medical Errors: survey questions

32. UK trainees reporting errors ‘in the last month’ Foundation compared to all other grades

33. F2 Trainees - Safe Doctors? Notable practice Barriers to good enough practice What needs to change?

34. F2 Trainees - satisfaction Felt subject to persistent behaviour undermining confidence and self esteem Considered daily leaving medicine %

35. F2 Trainees satisfaction Felt subject to persistent behaviour undermining confidence and self esteem Considered daily leaving medicine 10% 9%

36. Source of bullying – trainees who reported bullying only Cons Othr T’eeNrsMgr Pt/ rel Wk cultOthrMWN= All othr gde50%11%12%4%1%12%7%3%2,247 Fou ndat ion26% 25%2% 12%5%2%734

37. F2 Trainees - Satisfaction Notable practice Barriers to good enough practice What needs to change?

38. F2 Trainees Take Home Messages Notable Practice Potential barriers

39. F2 Trainees Notable Practice Action Plan Action Points for my workplace Useful contacts and resources

40. Response rates Trainee All usable responses = 33,329, Response rate = 66% where denominator is all valid trainee emails obtained via deanery returns and trainee opt-in form. Trainer GP = 2,211 (44%) Consultant = 7,810 (~20%) Total = 10,021 Analysis

41. The Reporting Tool Launched to deanery staff involved in the work 2 May for testing and comment prior to public launch. The reporting tool for the survey results is located here: http://reports.pmetb.org.uk Each type of report covers a different type of respondent group and is displayed via three levels of data, starting with the top level and going down to the lowest detail of individual items. Each level can be accessed by clicking through or selecting an option from the drop down list. Analysis

42. How To Use The Reporting Tool 1) After logging in, select the Group type you wish to view. Select the reporting group you are interested in. This includes selecting the survey type and year.

43. 2) Next refine your data selection, for example select the deanery and specialty of interest. How To Use The Reporting Tool To view the data for your deanery and specialty select the appropriate options in the drop-down lists.

44. 3) View each indicator score by scrolling down the page and note the national range in the green box, values will be shown when you roll your mouse over them. How To Use The Reporting Tool To view the next level of data please click here.

45. 4) The next screen displays mean indicator values within the selected group of providers. How To Use The Reporting Tool To view the deanery labels roll over each point To view the next level of data click on the yellow dot (the selected deanery) or select the deanery from the drop- down below.

46. How To Use The Reporting Tool To view the data labels for the bar charts roll you mouse over each bar. 5) The last screen displays individual items and also displays the N number. There is a select box on this page at the bottom to view other groups of items at this level.

47. Trainee Survey example All Trauma and orthopaedic surgery UK-wide 2007 This example shows a outlier below the national mean and in the lower quartile. In this case trauma and orthopaedic surgery F1, F2, FTSTA and Lower (ST1-3) trainees have outlier status for the overall supervision indicator score when compared to all trainees UK-wide.

48. Trainee Survey example All Trauma and orthopaedic surgery UK-wide F2 2007 By clicking through to the next level you can see that trauma and orthopaedic F2 score the lowest when compared to other surgical specialties for overall supervision.

49. Trainee Survey example All Trauma and orthopaedic surgery UK-wide F2 2007 Overall Supervision N = 240 The bottom level graphs show the items used to calculate the indicator. In this case the poor indicator score is partly attributed to 40% of respondents selecting ‘weekly’ to the statement shown on the right.

50. Trainer Survey example South Yorkshire & South Humber 2007 This indicator is the percentage of those trainers who perform workplace based assessments who have been trained to perform them. A high score indicates more trainers having received training in workplace based assessments out of those respondents who say they undertake this work. This example shows that the SYSH deanery have an above outlier, indicating that the mean score for these trainers is above the national mean.

51. Trainer Survey example Comparison across selected group of providers. South Yorkshire & South Humber The next level shows that the SYSH deanery has the highest % of workplace based assessment trained trainers in the UK. All deaneries. Workplace based assessments (%)

52. Trainer Survey example South Yorkshire & South Humber 2007 Feedback To Trainees (%) N = 210 The bottom level graphs show the items used to calculate the indicator. In this case the high indicator score is attributed to 75% of respondents stating that they have been trained in workplace based assessments.

53. The National reports for the Trainee and Trainer Surveys, 2007 will be launched week commencing 21 July 2008 Deaneries will respond to survey findings via the action plans contained within the deanery annual reports Launch event will be held in Cardiff 25 July 2008. National report

54. For additional information please see the PMETB website: www.pmetb.org.uk/pmetbsurveys If you have any comments or queries please contact: trainee.survey@pmetb.org.uk Or trainer.survey@pmetb.org.uk Further information