Supporting GPs to interpret and learn from Patient and Colleague Feedback Dr Di Jelley Northern Deanery Associate Advisor for Appraisal and Revalidation Deanery Conference 2012

What is Multi-source feedback [MSF] ? MSF provides the opportunity for patients, non- medical co-workers (including other health professionals, managers and administrators) and medical colleagues (including trainees and juniors) to reflect on the professional skills and behaviour of a doctor. GMC 2011

What is multi-source feedback for? Seen as a developmental tool to examine behaviours key to a job such as teamwork, communication and interpersonal skills Aim is to provide a set of colleagues’ and patients’ opinions based on GMC criteria, and to compare these with Dr’s self assessment Looks less at what people do, and more at how they do it Aim is to enhance individual and team performance

Experience of MSF in the health service Used extensively in North America as part of re- accreditation Can achieve high levels of reliability if raters are used Allowing Dr to choose raters does NOT lead to more +ve feedback than random selection Already widely used in junior Dr training Several feedback tools have now been evaluated in UK general practice

GMC feedback questionnaires-research findings Campbell et al report via GMC guidance on pilot work with > 17,000 colleague and patient responses - uk.org/executive_summary_of_research.pdf_ pdf GMC work and other studies have shown that accurate feedback on Dr performance can be provided not only by other Drs but also by nurses, other health professionals and administrators

Potential Benefits of feedback You don’t know how others see you until you ask Praise, and positive feedback, are both powerful motivators Can be a very powerful a learning experience Self assessment of communication skills, team relationships, leadership skills etc is very inaccurate

Possible Pitfalls with 360 degree feedback Destructive feedback can be very damaging to the individual and the team Anonymity may be used to express grudges or personal animosity Feedback without facilitated discussion unlikely to change behaviour Time consuming and respondents may get ‘feedback fatigue’

What tools are to be used for Patient and Colleague Feedback The RCGP Guide [version 7 June 2012 pp ] only recommends GMC Questionnaires, CFEP and Edgecumbe as tools for both Colleague and Patient feedback –other tools for either/or The surveys must be independently collected and collated They must be developed and piloted in line with GMC Guidance

GMC Questionnaires These have been developed and piloted by CFEP The questionnaires themselves are free to download from the GMC website,but they must be collated independently and provide the Dr with comparisons to national norms CFEP are one of the organisations who collate them for doctors [at a cost]-Clarity and the RCGP also include them in their e-portfolios Collation can also be done by the PCT or Foundation Trust

CFEP -CFEP uses both the GMC questionnaires and their own version of feedback tools-they are very similar but only the latter has locum specific data at present. CFEPhttp:// eral-practice/360.aspxhttp:// eral-practice/360.aspx

Edgecumbe and Clarity Edgecumbe use their own surveys which are similar to the GMC ones –RCGP approves both their colleague and patient surveys for Revalidation Edgecumbe doctor-360.php doctor-360.php Clarity offers GMC questionnaires as part of their annual fee, but they do not yet have much of a comparative data base

Interpreting patient and colleague feedback –some general points Responses from Patients Bias towards the positive –majority of all responses ‘good ‘ to ‘excellent’ only 1% ‘less than satisfactory ‘ or ‘poor’ 98% happy to see that doctor again Responses from Colleagues Again tend to be very positive –majority ranking their colleague as ‘good’ to ‘excellent’ Only 1% ranked their colleague as ‘less than satisfactory’ or ‘poor’ 97% agree their colleague was fit to practise medicine

Bench-marking [1] The data base of responses from the pilot studies is from volunteers who are unlikely to be a truly representative sample of all UK doctors The differences between the lowest and highest quartiles is often very small –this may increase as data base expands to include all doctors Appraisers should also be aware of rating biases when interpreting the doctor’s data

Bench-marking [2]- Rating biases Patient factors- the following lead to higher ratings: –Perceived importance of the consultation – –Well-established doctor/patient relationship –Ethnic group –responses higher from white than ethnic group patients –Age -patients over 40 Colleague factors- the following lead to higher ratings –Greater frequency of contact between doctor and rater –Non medical peers rate more highly than medical peers

Bench-marking [3]- Rating biases Doctor factors –the following score less highly overall from patients and colleagues –Locums –Doctors whose primary medical degree is from outside the UK Ideally the doctor should be compared with a cohort of patients/colleagues with similar characteristics, but this data is not yet widely available. CFEP does have some locum specific data with their own survey, not the GMC questionnaire yet

Understanding the feedback–CFEP and Edgecumbe present it differently Edgecumbe give a % value for every question –if the doctor’s responses are marked as 10% this means only 10 % of doctors had responses to this question which scored lower than this doctor’s response. If the doctor has a score of 85% for a particular question, this means that only 15% of doctors would have a higher score than this doctor on this question

Interpreting CFEP CFEP uses quartiles –the doctor’s responses are graded as ‘ lowest quartile -25%’ middle quartiles %- and top quartile –above 75% The concept is the same as Edgecumbe but the figures look a little different. If responses to a particular question are in the bottom quartile, this means they are the lowest 25% of scores-if the responses are in the top quartile then for that question the doctor’s responses are in the highest 25%.

Clarity and RCGP Use the GMC Questionnaires but not yet clear how the feedback will be presented to doctors At present both will have only a small data base of feedback responses, so any comparative data will need to be treated with caution This will improve as more doctors go through the feedback process

Interpreting the figures Given the research data findings that suggest most doctors are highly rated most of the time, it is worth identifying for discussion any areas where The responses are in the lowest quartile [CFEP] or below 25% [Edgecumbe] Any area where the range of responses is quite wide The responses are higher or lower than the doctor’s self assessment

Structuring the discussion about feedback Focus first on the overall results –most are likely to be good/very good overall and this should set the scene for the discussion Consider discussion about the process –How many questionnaires were returned –How and where was the patient survey done –Did the sample have any special characteristics –How many valid responses were received

Considering strengths and weaknesses Celebrate all the positives -high ratings and positive comments Do all the items get a similar rating or are there marked differences between items Does the doctor score significantly lower than other doctors [ie lowest quartile or <25%] on any items Does the self-assessment vary significantly from the patient/colleague scores Are there any comments that need specific review

General advice advice on how to discuss feedback Most doctors find eliciting feedback stressful and fear they will be ‘found wanting’-be sensitive to this and supportive in your comments –celebrating the positives The process is formative- NOT pass or fail –your aim is to make sense of feedback and use it to inform to shape the doctor’s CPD as needed Use open questions, identify strengths, development areas, and any unhelpful behaviours Should lead to agreement on what to keep doing and what to do differently Finish by summarising strengths

Consider the PINs model for supporting change-solution focused Platform for change (P) Describing the ideal (I) Where are you now? (N) Next steps (S)

Platform for change (P) [Dr Steve Blades] What is it important for you to change / improve? What will be the benefits of the change e.g. to you, to colleagues, to patients What would be the positive knock on effects of the change? How important is it to change on a scale of 1-10

Describing the ideal (I) Imagine you were doing this really well. What would be happening? What would you be doing? What would others see and hear? (Concentrate on the observable not on feelings)

Where are you now? (N) On a scale of 1-10 where 1 is never doing any of this and 10 is doing it all of the time where are you now? What makes it n? (rather than less than n) When are you able to do better than this? Take the opportunity to be affirming about the ability to accomplish the change at least partly or some of the time.

Next steps (S) What would be happening if you were doing n+1? What would you be doing? What would others see and hear? What small steps can you take to get you to n+1? How can you experiment with changes? What and when will you actually do? Who can support you in making these changes? What will be the first signs of change?

Summarising the discussion [1] When considering areas for change it may be helpful to think about some of these points What might be the benefits of change? These might be benefits for the doctor, patients or colleagues.

Summarising the discussion [2] Be as specific as possible about the change. For example if the doctor needs to communicate better with staff be clear about what is meant by this and how it might be achieved and reviewed? Making changes in this area can be difficult so be realistic about how many things can be changed at once

PDP aims –make them specific For some changes further training might be indicated. Actions might include attendance at a communication skills update, doing some joint or videoed consultations, specific training on shared decision making skills, leadership skills, mentoring, coaching etc Try and include a specific action on the PDP not just a vague aspiration

Self-assessment of your feedback review skills The University of Exeter Medical School are currently developing an on-line self-assessment tool for appraisers [using CFEP/GMC questionnaires] Collated feedback for a range of doctors will be available to view and assess Appraisers will be able to see how their interpretation of the feedback compares with that of other raters This will be a valuable education tool for appraisers and may help to identify appraisers who assess at the ‘hawk’ or ‘dove’ ends of the spectrum, and move them towards the middle ground

“Conclusion Given the broad range of ways peer evaluators can be used and the sizeable number of competencies they can be asked to judge, generalisations are difficult to derive and this form of assessment can be good or bad depending on how it is carried out ” Norcini, Med Ed 2003; 37: