Using 360-degree feedback as part of Senior Medical Performance Review in a public hospital setting Dr Dale Thomas Director Medical Services Redcliffe Hospital

Overview Describe the introduction of Senior Medical Performance Review including online 360-degree feedback tool Describe the introduction of Senior Medical Performance Review including online 360-degree feedback tool Discuss some pitfalls and challenges in how such processes could be adopted more broadly Discuss some pitfalls and challenges in how such processes could be adopted more broadly

Performance Appraisal Well established processes for most professional groups, including junior doctors, but not for senior doctors Well established processes for most professional groups, including junior doctors, but not for senior doctors Assessing trainees vs expert clinicians Assessing trainees vs expert clinicians Permanent vs temporary employees Permanent vs temporary employees Maintenance of standards vs continuing professional development Maintenance of standards vs continuing professional development

Clinician Engagement Senior medical staff are a valuable resource, but not often given feedback on their performance Senior medical staff are a valuable resource, but not often given feedback on their performance Doctors are competitive, high achievers Doctors are competitive, high achievers Even experts can improve their performance Even experts can improve their performance

Senior Medical Performance Review Clinical indicators – complications, re-admission rates, length of stay Clinical indicators – complications, re-admission rates, length of stay Participation in CME/CPD/MOPS Participation in CME/CPD/MOPS Participation in departmental meetings & clinical audit Participation in departmental meetings & clinical audit Records of mandatory training Records of mandatory training 360 degree feedback summary 360 degree feedback summary

360 degree feedback tool Based on a validated questionnaire 1 Based on a validated questionnaire 1 Developed following extensive consultation with AMA and unions 2 Developed following extensive consultation with AMA and unions 2 Anonymous online tool or paper survey Anonymous online tool or paper survey 1.Ramsey PG, Wenrich MD, Carline JC, et al, Use of peer ratings to evaluate physician performance, JAMA 1993 Apr 7;269(13): Johnston A, Senior medical performance review: making it happen – the Queensland experience, RACMA Quarterly 2011 Feb, accessed online performance-review-making-it-happen-the-queensland-experience&catid=21:the-quarterly-february- 2011&Itemid=14 performance-review-making-it-happen-the-queensland-experience&catid=21:the-quarterly-february- 2011&Itemid=14

360 degree feedback tool 9-point Likert rating scale 9-point Likert rating scale –Clinical domain 9 questions –Team management 4 questions –Interpersonal skills 5 questions –Ethical behaviour 4 questions Free text strengths & weaknesses Free text strengths & weaknesses

Implementation at Redcliffe Hospital Pilot study in Anaesthetic Department Pilot study in Anaesthetic Department –First cohort of 5 anaesthetists in 2010 –Second cohort of 6 consultants in 2011 –Third cohort of 5 anaesthetists in 2013 Physician cohort of 6 consultants in 2012 Physician cohort of 6 consultants in 2012 Stratified design allowing subgroup analyses Stratified design allowing subgroup analyses

Sampling methodology 1 Allowing subjects to choose who to invite to provide feedback Allowing subjects to choose who to invite to provide feedback Completed by consultant peers, supervisors, training registrars and nursing/support staff Completed by consultant peers, supervisors, training registrars and nursing/support staff Collated feedback provided to SMO/VMO for discussion with their Clinical Director Collated feedback provided to SMO/VMO for discussion with their Clinical Director

Sampling methodology 2 Consistent sample for each subject Consistent sample for each subject Respondents Respondents –Supervisor/line manager –Consultant anaesthetists –Anaesthetic registrars –Nursing and allied health staff –Proceduralists – i.e. consultant surgeons, obstetricians, gastroenterologists –Optional self-assessment

Response Rate Managers 94% Managers 94% Peer consultants 74% Peer consultants 74% Registrar trainees 57% Registrar trainees 57% Nursing/support 76% Nursing/support 76% Proceduralists 40% Proceduralists 40% Self-assessment 81% Self-assessment 81% Total excluding self 60% Total excluding self 60%

Results Higher response rate for subjects who are Staff Specialists (72%) compared with VMO/SMO (44%) Higher response rate for subjects who are Staff Specialists (72%) compared with VMO/SMO (44%) Overall very good quantitative scores from all groups of respondents Overall very good quantitative scores from all groups of respondents

Qualitative 75.5% of respondents provided comments 75.5% of respondents provided comments 63% of comments were positive 63% of comments were positive Negative comments can be confronting, but were considered more useful by subjects Negative comments can be confronting, but were considered more useful by subjects

Feedback to subjects Quantitative & qualitative results Quantitative & qualitative results Summary data included as part of annual PAD process Summary data included as part of annual PAD process Inform professional development needs for coming year Inform professional development needs for coming year

Pitfalls & challenges Choosing your respondents Choosing your respondents Online confidentiality concerns Online confidentiality concerns Lower response rates for VMO and non-specialist SMO Lower response rates for VMO and non-specialist SMO Meaningful subgroup analyses Meaningful subgroup analyses Evaluating effectiveness Evaluating effectiveness

Questions?