1. Bob Woodwards SAC Chair, Oral and Maxillofacial Surgery

2. What do we want?  Objectivity  Reproducibility  Fairness  Ability to spot underachievement early in training path

3.  Competence-based curriculum  Assessment framework  Syllabus - explicit standards for common surgical and specialty-specific knowledge, clinical judgement, technical and operative skills and generic professional skills.

4. Does Shows how Knows how Knows Assessment in simulated settings Assessment in the work place Miller GE. Acad Med 1990;65 (Suppl.):S63–S67.

5. Producing the “Surgical Expert”

7. Key roles The Programme Director The Assigned Educational Supervisor Clinical Supervisor Trainers – in the multi-disciplinary team The trainee

8. Trainee-led learning Guided by the Assigned Educational Supervisor Making meetings with the AES Uploading assessments to portfolio Completing the Learning Agreement Triggering assessments, engaging assessors Reading guidance notes Taking up learning opportunities (planned or ad hoc) Being prepared in good time for reviews

9. Principles of WPBA Assessments for learning The main purpose is to provide feedback Start early in the placement and spread throughout year Fits into normal practice Repeated to show progress Different settings, patients, different assessor improves reliabilty Trainees upload all assessments – good and less good

10. Feedback Should cover: How the trainee felt about the performance Trainee’s strengths Suggestions for development Action plan for improvement

11. Steps in workplace assessment Observation fits into normal practice Provides verbal constructive feedback immediately after Completion of form (if paper-based keep a copy) Trainee uploads assessment into the portfolio Assessor validates the assessment

12. ISCP WPBA methods Mini-Clinical Evaluation Exercise (Mini-CEX) Case-based Discussion (CBD) Direct observation of procedural skills (Surgical DOPS) Procedure-based Assessment (PBA) Multi-source feedback (revised version of Mini-PAT coming soon)

13. Clinical Evaluation Exercise - Mini-CEX Trainee-patient encounter e.g. A&E, clinic, ward Observation + 10 mins for the form + feedback History taking, physical exam, management, communication At least 6 per year Standard set is for the completion of that stage Assessors are consultants, senior trainees, staff grades, other healthcare professionals + one by the AES

14. Case Based Discussion - CBD Focussed discussion about a challenging case managed by the trainee Based on trainee’s entries in patient’s notes 30 minutes e.g. as part of mid-point appraisal Ideal for evaluating reflective practice 6 per year absolute minimum Standard set is for the completion of that stage Assessed by trainee’s AES or equivalent

15. Direct Observation of Procedural Skills - Surgical DOPS Covers all skills required to successfully perform a simple procedure Ideally whenever a procedure is carried out Observation + 10 mins for the form + feedback Assess at least monthly Assessors are consultants, senior trainees, staff grades, other healthcare professionals + one by the AES

16. Procedure Based Assessment - PBA Covers all skills required to successfully perform a specialty index procedure Ideally whenever a procedure is carried out Observation + 10 minutes for the form + feedback Assess at least monthly Set at the standard of CCT Assessor must be a consultant + one by AES

17. 11132 records from 2071 trainees global rating by level of training  ST12.28  ST2 2.61  ST32.81  ST43.06  ST53.34  ST6-8 3.69 After J Foulkes 2009

18. Multi-source Feedback - MSF Identifies problems of professional behaviour against GMP 1 per year, can be repeated if necessary Trainee self-assessment and 8-12 team ratings Health and probity issues can be raised anonymously Electronic feedback via the Assigned Educational Supervisor AES has a face to face meeting with the trainee to present feedback and sign it off

20. Utility Blueprinted against GMP Studies - JCST evaluation, PBA study by Jonathan Beard Valid - PBAs developed and trialled by SACs Valid – Methods developed and validated in Foundation Educational impact – quality of feedback and action plan, must be followed by reflection Feasible - fits into normal practice Reliable – different settings, patients and assessors Acceptability – trainer/trainee engagement Cost effective – time needed

21. Portfolio evidence A range of different types of assessments in different settings Assessors’ written comments The mini-PAT report by the AES The surgical logbook Trainees’ presentations, audits, projects, reflections Records of discussion with the trainee in appraisals Learning Agreement, in particular AES’s report

22. AES’s end of placement report Makes use of: Evidence in the portfolio, especially assessors’ comments Day to day observations of the trainee A debriefing session with the clinical supervisor Learning Agreement outcomes plus notes Standards of GMP

23. Preparation for the ARCP Deanery should provide at least 6 weeks’ notice to trainees It is the trainee’s responsibility to provide a portfolio of complete evidence Trainees should be aware that incomplete portfolios will result in outcome 5 Ensure ARCP panels are thoroughly briefed

24. ARCP Evidence Learning agreement, signing off syllabus topics AES reports WPBA in portfolio showing progress Exams Surgical logbook Audit, research, projects

25. ARCP Outcomes  Satisfactory Progress  1. Achieving progress and competences at the expected rate (clinical)  Achieving progress and competences at the expected rate (academic)   Unsatisfactory or insufficient evidence (trainee must meet with panel)  2. Development of specific competences required – additional training time not required  3. Inadequate progress by the trainee – additional training time required  4. Released from training programme with or without specified competences  Released from academic programme  5. Incomplete evidence presented – additional training time may be required   Recommendation for completion of training  6. Gained all required competences (clinical)  Gained all required competences (academic)

26. www.iscp.ac.uk