Educational Supervision & Find Your Way Around in the E-portfolio Dr Jane Mamelok RCGP WPBA Clinical Lead

Workplace Based Assessment “The evaluation of a doctor’s progress over time, in their performance in those areas of professional practice best tested in the workplace” FORMATIVE Summative judgment made by panels informed by evidence from WPBA This is how WPBA was defined for the PMETB submission. The important point to recognise is that the training programme is mapped to the curriculum objectives and WPBA is the opportunity for the trainee to gather evidence and build their portfolios over the entire training programme. The Educational supervisor has an important role in guiding them through that process. They must understand the programme, the necessity for it to be rooted in the curriculum and the assessment schedule and requirements for each trainee. This point may be developed later in the presentation and discussed in the group. The process is formative, gathers evidence over time that informs the panel process and the summative judgment.

Miller’s Pyramid Knows Knowledge AKT Shows How Knows How Performance Competence CSA Does Action WPBA To remind you where it sits on Millers pyramid of competence, WPBA because it looks at performance in action in the workplace tests higher up Millers and offers the opportunity if it tests across a range of complexities it can also push up Bloom's taxonomy. AKT sits at the bottom, knows and knows how. CSA pushing higher up WPBA because it relates to the learners’ experience in the workplace sits at the top

Coverage by assessments Curriculum WPBA CSA AKT Primary care management   Person centred care Specific Problem-solving skills Comprehensive approach Community orientation Holistic approach Contextual Aspects Attitudinal Aspects Scientific Aspects Reminder of the blueprint areas for the components This is useful as it demonstrates that WPBA crosses nearly all the blueprint areas and also addresses the gaps not covered by other components. Credit for this slide is Amar Rughani and the blueprint. Psychomotor skills

Working with Colleagues Practising Holistically Maintaining an Ethical RELATIONSHIP DIAGNOSTICS Communication & Consulting Skills Working with Colleagues & in Teams Practising Holistically Data Gathering & Interpretation Clinical Management Managing Medical Complexity Making a Diagnosis / Making decisions Professionalism Maintaining an Ethical Approach to Practice Fitness to Practise WPBA is underpinned by a competency framework derived from the curriculum domains and mapped to good medical practice. This slide usefully categorises them into 3 areas Relationships Diagnostics Management & Professionalism The slide should be attributed to Tim Norfolk A useful ice breaker in a group us to delay the slide which is animated and ask them to name the 12 competency areas – most people can get about 8 or 9 with some prompting. I usually swot up beforehand so as not to embarrass myself. Maintaining Performance, Learning & Teaching Primary Care Administration & IMT Community Orientation MANAGEMENT

The potential of linking WPBA & the learning portfolio. WPBA assesses performance in the workplace over the entire training programme against 12 defined competencies. The portfolio gathers the information and offers the opportunity for reflection and formative feedback. Links assessment to learning outcomes and continuing professional development.

Clinical Supervision Clinical supervision describes the framework for regular, structured encounters reflecting on casework in the context of the post or specialty in which the health professionals are working and aims to identify areas of best practice and developmental needs. It has an important clinical governance function. Clinical supervision Clinical supervision has been used in mental health services for many years, and describes the framework for regular, structured encounters reflecting on casework in the context of the post or specialty in which the health professionals are working and aims to identify areas of best practice and developmental needs. Clinical supervision has an important clinical governance role setting and improving professional standards of care. Clinical supervision can be a one to one encounter or between groups of practitioners. In the context of specialty training, clinical supervision usually takes the form of one to one encounters between trainee (supervisee) and a more senior and experienced clinical supervisor.

Educational supervision Educational supervision is organised supervision taking place in the context of a training programme, it is aimed more to act as an umbrella to guide the trainee through the training programme than assessing and discussing individual cases (which is in the domain of the clinical supervisor). Educational supervision is organised supervision taking place in the context of a training programme, it is aimed more to act as an umbrella to guide the trainee through the training programme than assessing and discussing individual cases (which is in the domain of the clinical supervisor). Both clinical and educational supervisor may have conflicting roles. One is developmental, allowing reflection and development, the other to provide performance assessment. The summative nature of the end point of postgraduate training and the contribution clinical and educational supervisors’ report make to that final summative decision, means there is no easy way to resolve this conflict and it must be recognised and carefully managed.

Gold Guide Requirements Clinical supervision Each trainee should have a named clinical supervisor for each placement, usually a senior doctor, who is responsible for ensuring that appropriate clinical supervision of the trainee’s day-to-day clinical performance occurs at all times, with regular feedback. Gold Guide definitions The Gold Guide defines and differentiates the two roles as follows: Clinical Supervision Each trainee should have a named clinical supervisor for each placement, usually a senior doctor, who is responsible for ensuring that appropriate clinical supervision of the trainee’s day-to-day clinical performance occurs at all times, with regular feedback. Educational supervision Educational supervisors are responsible for overseeing training to ensure that trainees are making the necessary clinical and educational progress. Where possible, it is desirable for trainees to have the same educational supervisor for the whole of their training programme or for stages of training (e.g. the early years or more advanced years of training).

Gold Guide Requirements Educational supervision Educational supervisors are responsible for overseeing training to ensure that trainees are making the necessary clinical and educational progress. Where possible, it is desirable for trainees to have the same educational supervisor for the whole of their training programme or for stages of training (e.g. the early years or more advanced years of training). Gold Guide definitions The Gold Guide defines and differentiates the two roles as follows: Clinical Supervision Each trainee should have a named clinical supervisor for each placement, usually a senior doctor, who is responsible for ensuring that appropriate clinical supervision of the trainee’s day-to-day clinical performance occurs at all times, with regular feedback. Educational supervision Educational supervisors are responsible for overseeing training to ensure that trainees are making the necessary clinical and educational progress. Where possible, it is desirable for trainees to have the same educational supervisor for the whole of their training programme or for stages of training (e.g. the early years or more advanced years of training).

What makes a good educational supervisor ? Senior educator with an understanding of training programme, assessments and employment issues. Good listener offering supportive challenge. Can give constructive feedback Professionalism, recognises the conflicting roles, sets boundaries etc. What are the qualities and attributes of an educational supervisor? The ES should be a senior person with knowledge and understanding of the educational programme, assessments and employment issues in the context of the trainees’ professional development journey. A good listener, with counselling skills and analytical, critical thinking. Understand feedback principles and be able to ask discriminating questions, giving constructive feedback, supportive challenge, allowing formative development. Non-judgmental, the relationship relies on mutual trust and respect and therefore the ES should provide non judgmental reflection of observed behaviours and be able to set the necessary boundaries when assessment of performance in practice is required. Self awareness and insight into own weaknesses and potential pitfalls. The ES should be able to recognise the problems created by transference and dependence the ES/trainee relationship can create and have strategies for preventing stress and burnout. Professionalism, the ES must treat this relationship with professionalism, respecting confidentiality, making the process transparent. Motivation, the ES should want to be involved in this privileged relationship however, educational organisations should value its important role in professional development and it should be appropriately rewarded.

The role of the ES in GP specialty training. This can be summarised as providing an overarching umbrella of guidance helping the trainee get the best out of their programme and navigating their way through it. However, the ES assesses the rate of progress and development of competence contributing to a summative judgement. The primary role of the ES is to guide the trainee through the programme and get the best out of it. The problem is that they do have a responsibility to monitor progress through the programme and assess how trainees are progressing through the programme and their progress in development of the required skills and competencies. The ES reports and their recommendations contribute to the final summative judgement.

Varied roles of the Educational Supervisor Guidance and support with performance appraisal Learning needs assessment, empowering and facilitating the learner. Conduit and network function Appraiser and counsellor Coaching and career counselling Guardian role The key roles for an Educational Supervisor can be summarised (4). An ES provides a guiding and supportive role; it cannot be divorced from performance appraisal and the potential conflict should be recognised and managed in such a way that does not discourage openness from the learner. Guide learners through their programme, identifying learning and development opportunities. Contact and conduit function, the ES can provide regular contact outside of the programme and act as a conduit for resources and liaison with programme staff. Information and network resource, signposting the learner, “opening doors” and providing introductions appropriately. Counsellor, it is the role of the ES to listen, provide constructive and challenging reflection and facilitate problem solving. To nurture, identifying strengths and weaknesses, empowering and facilitating the learner on the journey from novice to expert. Provide an appropriate role model. Coaching, there is a role to coach learners through key stages, grooming for interview etc. Finally there is a guardian role, the ES can pick up potential problems and provide an understanding of the support and referral framework allowing both ES and trainee to fulfil their professional obligations in Good Medical Practice (5) (e.g. encouraging a sick doctor to seek help).

So you are supervising a GP specialty trainee, what do you need to do? Identify you learner, ST1 ST2 or ST3. Arrange an early meeting to set boundaries, frequency of meetings etc. The ES needs to be able to navigate the e-portfolio proficiently. Review the evidence. Give formative feedback and define PDP at the staged reviews. Discuss the difficulty of managing more than one supervisee, most of us have 1 or 2 at each level. Setting boundaries is important – some trainees expect the ES to read and validate all their entries. You might want to discuss the differing roles of the clinical supervisor in conducting the assessments (or delegating them) and the ES who review them and assesses progress and feeds that back. Being able to navigate the EP is hugely important and they MUST have access in the ES role as well as trainer to be able to access all the relevant pages and to see and write in the review page.

Reviewing the evidence Check assessment schedule. Review the self assessment ratings. Review PDP objectives if follow on review. Review competency and curriculum coverage Purposeful sampling of the learning log – descriptive v reflective. Look at CSR is there evidence to support their judgement? Hone in on some of the COTs and CBDs

The Review page e-portfolio This page demonstrates that they need to be logged on as an ES and to use the review section – red arrows

Check the assessment schedule Click on all reviews and that gives the up to date summary of assessments and evidence

Review the PDP This is an example of the PDP entered by the trainee

Review curriculum & competency coverage and self rating scales This the trainee self assessment rating which the ES can hone in on and make comments

Sample the learning log The learning log, again click on the magnifying glass allows you to hone in, entries need to be shared and then sample and read.

Review the Clinical Supervisors report This is an example of a poor CSR where the trainee has been graded as competent across all the competencies and there is no free text from the CS to justify their judgements, this makes it very difficult for the ES to make a judgment on progress.

Questions for debate How much is enough? How much evidence do you need to make a judgement? What do we mean by competence? Lesley Southgate makes the point that you only need to see poor performance repeated 2 -3 times to make a judgement that someone is not fir to progress but more is required to make the borderline/ competent decision. Competence in this context means competent for licensing and independent practice NOT against peers or for stage in training. This usually proves contentious and a good point to develop if you use group work.

How much evidence is enough? The evidence gathered from WPBA builds up a “picture” of the competent GP. What WPBA is doing is building up a picture based on evidence of the competent GP. Competent for licensing, not at the stage of training or measures against their peers. If we think of the evidence as pixels in the picture you can see that the more pixels (higher the definition) the clearer the picture becomes. The debate rages on about how clear the picture needs to be (what is the minimum evidence required) to decide when the picture is complete and when a GP can demonstrate competence.

Talk through building the picture up Talk through building the picture up. When you have the clear picture flip between the mid way and end point.You could recognise Steve at level 2 so is that as much evidence or do you need the clarity of level 3.

Defence of evidence Have we got this right yet? Medical Schools using portfolios for undergraduates have a portfolio interview which allows the trainees to defend the evidence and this informs the portfolio assessment decision. Is there a role for portfolio viva? Is that final ARCP panels Or RCGP oral examiners? You might want to include this to discuss making judgements and portfolio assessment robust. I stress that this is my personal view as lead and not RCGP policy but if we had portfolio interviews we would have a World Class Portfolio assessment in the same way as CSA is being heralded as World Class. Whilst I can be that provocative you may chose not to.

Assessment decision Portfolios are criticised because they depend on subjective judgements that are unreliable. Schuwirth (2002) argues that they can be reliable if but the “judgements must be collected in a way that samples through possible sources of bias” – sampling professional behaviours on multiple occasions and multiple assessors

Key steps for development Define a portfolio framework but no so restrictive as to trivialise the evidence . Encourage reflection – can be learned. Educator and assessor training are crucial. Need to understand competency framework Define competence Achieve acceptable inter rater reliability Encourage Balanced portfolios Robust WPBA and portfolio standards for portfolio assessment – defend the JPC decision. Developments in e-portfolio functionality must match the work on standards and competency framework. This is a useful summary slide to inform of the changes the WPBA have planned. The key point is that educator support and training a crucial as is the development of the competency framework that underpins WPBA. Without an EP with a specification and functionality that delivers on the archive and ANLYTICAL role, that is intuitive and easy to use any developments in WPBA will not work.

Summary The educational supervisor has a key role in directing and supporting the training programme for each learner. There are useful analogies between the role of the appraiser and the appraisal portfolio and the similar role for educational supervisors. Training and support for trainees and educators are key. The e-portfolio must be user friendly and functionality appropriate to purpose. Your feedback (to WPBA group) is important. Email WPBA Lead j.mamelok@nwpgmd.nhs.uk I like the analogy that using the EP should be as easy as buying theatre tickets on line and not dependent on umpteen PDF user guides. I make the point that I need their feedback as users for change and am happy for emails at the deanery.