The Royal College of Pathologists – the overall assessment system Dr. Trevor Gray Director of Examinations and Assessment

The Assessment System Overview of PMETB and assessment strategy How WBA fits with other assessments Advantages and disadvantages of WBA compared with other assessments Local organisation of WBA and how it is administered by the College Future prospects

Overview PMETB (and the MMC process) has laid down significant challenges to medical postgraduate training (elegantly dissected in the Tooke report) PMETB principle is that training and assessment should be much more integrated – an assessment system Workplace-based assessment is an essential element of the assessment of trainees undertaking specialist medical training The specialty curriculum, training, exams and assessments are core College business (in the charter) The College’s aim is to develop a validated assessment system which is fit for purpose

PMETB strategy on assessment PMETB principles for assessment 2004 Nine principles for assessment systems Now backed up with technical documentation Principles based on “overarching strategy” Workplace-based assessment Examinations of knowledge Examinations of Clinical Skills (“Workplace-based assessment” PMETB, Jan 2005) Specialty curriculum and assessment system now approved by PMETB (i.e. match strategy)

Workplace-based Assessment Assessment based on performance in the workplace and measured there Assessed locally by supervisors as well as staff of all grades in contact with the trainee Assessment occurs during normal work immediately followed by feedback so trainee receives continuous feedback on their routine performance Results put into trainee portfolio for their annual review and hence important for trainee development and ultimately their employment contract

College Assessment System approved by PMETB Workplace-based assessment Year 1 assessment MRCPath Part 1 (test of knowledge) MRCPath Part 2 (test of skills and applied knowledge) Feedback on regular basis (WBA) Educational supervisor’s report ARCP (Annual Review of Competency Progression)

The College Assessment system: Miller’s pyramid Knows Knows how Shows Does MRCPath Part 1 MRCPath Part 2 Practical WBA e.g. Mini-CEX, DOPS WBA - MSF

Value of multiple assessments Key is several modes of assessment – triangulation A single exam system (MRCPath) does not provide adequate coverage of clinical practice WBA is also not sufficient in itself (Assessment System Guidance – PMETB)

Examinations and Assessment Examinations are simply one particular form of assessment Assessments are: formative (involve feedback and are intended to guide training) or summative (assess whether reach given standard) e.g. driving test, MRCPath All assessments should be “blueprinted” to the curriculum to demonstrate that the most appropriate method is being used

How Workplace-based Assessment fits with MRCPath Workplace-based assessment depends on assessment by local staff, examinations rely on assessment by those who have had no contact with the trainee Examinations provide an independent assessment of knowledge and practical skills to national standards Workplace-based assessment provides a local insight into candidates’ performance in routine tasks and interaction with staff and patients

How Workplace-based Assessment fits with MRCPath Workplace-based assessment is a local process governed by deaneries albeit to PMETB standards Examinations are organised nationally by the College Comments, complaints and appeals for WBA go to deanery, for exams go to College

Workplace-based Assessment – Advantages Measurement occurs in more authentic environment than the simulated one of the examination room Trainees can take assessment in their own time and are not limited to exam timetable The assessment tools are set up to provide extensive feedback to the candidate (formative) Use of computerised forms makes them easy to record

Workplace-based Assessment – Disadvantages To avoid time-consuming “busy” assessment forms, the assessments are often simplified if not simplistic A robust rating can only occur if there are a number of independent observers – which can be difficult in small specialties They have only been validated in a number of clinical settings but not in all pathology settings Computerised forms can over-simplify complex judgements and may not reward excellence appropriately

Year 1 Assessment (OSPE) Developed to provide evidence of progression in run-through training in the absence of further recruitment and selection Designed to assess progress in first year and be taken after 7 months training It is a test to “indicate the capability and potential of a trainee through a test of applied knowledge and skills relevant to the year 1 curriculum”

F1F2ST1ST2ST3ST4ST5 WBA Y1A MRCPath CCT Part 1Part 2 MSF Workplace-based assessment ARCP Foundation Stage AStage B Stage C Stage D Training and Assessment Plan

The training environment Workplace-based assessment utilises the skills of a wide variety of professional groups as assessors: medical, nursing, BMS, clinical scientist, ancillary staff The College cannot train or quality assure all assessors Training centres are recommended to develop a training environment where those with the skills needed for assessment are recognised and training is cascaded from those attending events such as this

Workplace-based Assessments and the College WBA is part of assessment system for Specialist Trainees from 1 st August 2007 Assessments need to be undertaken from early in training and recorded for ARCP 2007 – Launch of tools and paper-based assessment forms College is researching web-based systems similar to the ones used in foundation

Workplace-based Assessments and the College April 2008 – pilot web-based system By August 2008 – launch of web-based system Use data generated by all assessments including MSF and OSPE for evaluation of assessment strategy Ensure compliance with MMC Gold Guide and planned ARCP

Vision for College Assessment system College will have integrated specialty curriculum and assessment system endorsed by PMETB College will have fully validated workplace- based assessment tools delivering feedback on progress to trainees throughout course Valid and reliable tests will underpin MRCPath examination – Part 1 test of knowledge, Part 2 test of applied knowledge and skills

Factors affecting future assessment strategy Selection into specialty may not be an annual process in future compromising ability to deliver year 1 assessment (except maybe in Histopathology) Multiple recruitment may require more flexibility in arrangements for WBA Selection into specialty may involve a national set of tests (Tooke and Academy) displacing Year 1 assessment

Factors affecting future assessment strategy Workplace-based assessment is likely to continue but its use must be evidence-based Workplace-based assessment is not cast in stone – work in progress and subject to improvement following evaluation Place of individual components of assessment in MRCPath, Year 1 assessment and workplace-based assessment may change

Summary The College’s education and training, assessment and examinations departments have worked hard to produce curricula and assessments acceptable to PMETB This training day is the first for workplace-based assessment and is intended to assist training centres to develop assessment locally Workplace-based assessment will be subject to a full programme of validation and optimisation to ensure fitness for purpose The College is looking to source a web-based system to support workplace-based assessment

Glossary ARCPAnnual Review of Competence Progression CbDCase-based Discussion DOPSDirect observation of practical skills ECEEvaluation of clinical events MSFMulti-source feedback (ePATH-SPRAT) Mini-CEXMini clinical evaluation exercise OSPEObjective structured practical examination PMETBPostgraduate medical education and training board WBAWorkplace-based assessment