Workplace-Based Assessment Making it More Effective Jonathan Beard Professor of Surgical Education University of Sheffield, UK

Objectives Check knowledge of Workplace Based Assessment (WBA) Understand the purpose of assessment Review some of the problems of WBA Discuss possible solutions Any other objectives?

Workplace Based Assessment Workplace Based Assessment “Assessment of day-to-day working practices undertaken in the working environment” WPBA now an integral part of medical training (competence-based curricula) Integrates teaching, learning, assessment & feedback Designed to assess skills rather than knowledge Competence best assessed in the workplace as closest to actual performance Primary aim is to aid learning

WBA Tools Mini-CEX DOPS Specialty tools –e.g. PBA (ISCP), ACAT (RCP) CBD Assessment of Teaching or Audit MSF: –Only tool that does not require rater training Opportunistic Planned Once per placement

What is the Point of Scoring WBA? It depends on the purpose of the assessment

Purpose of Assessment? To aid learning through constructive feedback: –Assessment for Learning (formative) –Done frequently To check knowledge or skill has been learned: –Assessment of Learning (summative) –Done infrequently

Time for Teaching and Learning How many hours of teaching and learning does a trainee get per week?

Learning to Drive Driving Lesson Assessment for Learning Driving Test Assessment of Learning

The Danger of Confusing the Purpose of WBA Doctors regard all assessments as tests Therefore they practice informally before requesting an assessment, choose easy tasks and make the request after the event Assessors feel pressured to give a good score More attention paid to score than feedback Primary purpose of WBA as an aid to learning is lost

Changing the Name Supervised Learning Events (SLEs) done frequently to aid progress –trainee chooses the topics and timing –No score but feedback must be documented Assessments of Performance (AoPs) done less frequently to check that key competencies have been achieved –Identified by the specialty curriculum –‘Satisfactory’ or ‘development required’

Placement Two Placement Three Level of Competence Training Rotation Frequency and Timing of WBA Placement One Supervised Learning Event Assessment of Performance Able to perform the task

Changing the Language Similar format for many existing assessments Norm-based rating scale: 1-6 where 4 = ‘meets expectations for completion of stage of training’ Standard not defined but can usually be found ‘somewhere’ in the syllabus

What does ‘Meets Expectations for Completion of a Stage of Training’ Mean? To the Trainee? To the Assessor? To the ES/TPD? To the Employer? To the Patient?

Evaluation of WBA in Surgery DOPS –1370 trainees (ST1-4), 6240 DOPS (m=3.4) –Most domains rated 4 or 5 (m=4.8) –No construct validity and no reliability Similar findings by RCP for CEX and CBD PBA (Procedure-Based Assessment) –Criterion-referenced rating scale –85 trainees (ST1-7), 749 PBAs –Good internal and construct validity –High reliability (G>0.8 for 3 assessments)

Redesign of WBA Tools Criterion-referenced with clear performance anchors (e.g. could perform the task without help) Use descriptors not scores (e.g. able to do the task) Emphasis on feedback (e.g. make documentation of feedback mandatory) Clearer guidance notes for trainees (e.g. timing and frequency) and assessors (e.g. seniority and training)

Revised Surgical DOPS (ISCP) Verbal feedback and documentation of feedback is mandatory

Revised Ratings for CBD and Mini-CEX (RCP)

Assessment Methods Can the same assessment method be used for both purposes? Yes - but only if its use on each occasion is agreed and documented And - feedback is documented if it is an SLE (remove all ratings?) And - reliability is high enough for an AoP (well trained assessors)

Example of Implementation for Surgical Specialties 40 SLEs per year (1 per week) 3 AoPs (PBAs rated at level 4) required for every index procedure before CCT Logbook will ask whether an SLE or AoP was done SLE feedback must documented by trainee and validated by assessor AoPs can only be done by CS or ES

Summary Postgraduate medical curricula will need to be amended in light of the GMC discussion document No sudden change required – direction of travel – but need to be considered when reviewing curricula Trainers and trainees need an update Don’t mention the ‘A’ word Ask the question: driving lesson or test?

Further Reading and Information Further reading: –Impact of workplace based assessment on doctors’ education and performance: a systematic review. Miller & Archer, BMJ 2010 –Changing the Name and the Language. Crossley and Jolly. Medical Education 2012 More information: – – – uk.org/Learning_and_assessment_in_the_clinical_environment.pdf_ pdf uk.org/Learning_and_assessment_in_the_clinical_environment.pdf_ pdfhttp:// uk.org/Learning_and_assessment_in_the_clinical_environment.pdf_ pdf

Some Questions Do we need separately designed forms for SLEs and AoPs? How many hours of teaching and learning should a trainee have per week? Should MSF be designated as an AoP? Should AoPs only be done if there is no better way of testing a skill/behaviour? How do we communicate the new terminology and philosophy to trainers and trainees?