RANZCP Competency-Based Fellowship Program – WBAs and EPAs John Crawshaw Chair, Board of Education, RANZCP RANZCP Congress, 2012, Hobart, Tasmania

Presentation Overview Introduction Formative & Summative Assessments Workplace-based Assessments – WBAs –Why use WBAs? –When to use WBAs? –WBA tools and how to use them Entrustable Professional Activities – EPAs –What is an EPA? –When to entrust an EPA –Standard of entrustment at Stages 1, 2 & 3 –How the EPA Handbook and RANZCP Certificate of Entrustment (COE) work –Process of entrusting an EPA How WBAs may help gauge readiness to entrust an EPA Questions

Introduction - Competency-based Fellowship Program Aims Develop a competency-based curriculum and Fellowship training program that -integrates the concept of competent performance into the College’s educational philosophy -integrates contemporary concepts of best practice in adult education into the training program -is developed in line with international specialist medical education best practice -increases flexibility in training -allows for more timely progression

Introduction continued The competency-based 2012 Fellowship training program –is developed to better reflect the complexity of contemporary professional practice of specialist psychiatrists (medical expert; communicator; collaborator; manager; health advocate; scholar and professional) –provides a broad range of clinical experience during training –addresses issues associated with competing demands for service provision and training requirements for trainees –provides for Recognition of Prior Learning (RPL) –maintains high standards of training Workplace-based assessments align to, and assess, the development of competencies

Formative Assessment – Why Use WBAs? Competency-based training requires actual demonstration of the knowledge, skills and attitudes needed for safe practice WBAs –measure what doctors actually do in practice –assess competence in an authentic setting –provide mechanism for structured feedback from supervisor to trainee –help assess trainee progress –encourage concrete and effective feedback –formalise and add structure to current supervisor activities

Summative Assessments – Why Use EPAs? Entrustable Professional Activities (EPAs) –are representative tasks that are only entrusted to competent professionals –are summative assessment tasks in the competency- based Fellowship program –capture and record supervisor judgements about trainee performance and competence

Formative & Summative Assessments FormativeSummative Assessment for learningAssessment of learning Immediate feedbackEvaluates trainee’s learning at a particular point Feedback identifies strengths Feedback: Pass/fail Percentage Brief comments Feedback highlights areas for improvement Must pass to progress Feedback typically delayed Ways to improve performance outlined Exams, Psychotherapies, Scholarly Project, ITAs WBAs EPAs

How Many WBAs and When? WBAs are a way to: –Provide feedback for future learning –Help track and assess trainee progress WBAs may occur –Early, midpoint and towards the end of a rotation –Whenever a trainee is seeking to gain formal feedback Three WBAs required before each EPA can be attained –But, three WBAs does not mean an EPA has been achieved

WBA Tools Four Board of Education approved WBAs: –Case-based Discussion (CbD) –Mini Clinical Evaluation Exercise (Mini-CEX) –Observed Clinical Activity (OCA) –Professional Presentation Trainees are responsible for –Initiating the WBA process –Retaining the WBA forms after completion –Updating their learning plans

WBA Assessment All WBAs are assessed on a 9-point standard rating scale Trainee StageBelow standard for end of Stage (1 2 3) Meets standard for end of Stage (4 5 6) Above standard for end of Stage (7 8 9) Stage 1 – Basic standard Below standard for Basic trainee At Basic level as described on developmental descriptors Above Basic level. Moving towards the standard of a Proficient trainee Stage 2 – Proficient standard Below the standard of a Proficient trainee. Meets standard of a Basic trainee. Meets the standard of a Proficient trainee. Above the standard of Proficient trainee. Moving towards the standard of an Advanced trainee. Stage 3 – Advanced standard Below standard for Advanced trainee. Meets standard of a Proficient level trainee. Meets the standard of an Advanced trainee. Above the standard for an Advanced trainee.

WBA Assessment continued The mid-point of the scale – ‘Meets standard for end of Stage’ – is the standard trainees should aim to achieve on completion of each Stage Assessor also provides feedback comments Not all assessed criteria will be assessed on each WBA form Choose the appropriate assessment criteria from those listed on the WBA form

WBA Assessment – Developmental Descriptors Developmental Descriptors: –are behavioural descriptors for the Fellowship Competencies –articulate how the Developmental Trajectory applies to the Fellowship Competencies through the training stages –that articulates what is expected at each training stage is provided for each aspect of practice –are used as a guide when assessing a trainee’s performance on a WBA

Example: Aspect of Practice: Assessment By the end of each Stage, the trainee’s performance in this aspect of practice can be assessed using the following standards: Use of Developmental Descriptors in WBA Assessment Stage 1 Basic level Stage 2 Proficient level Stage 3 Advanced level Conducts a standard assessment of a patient with typical psychiatric disorders, but requires supervision to elicit all necessary data and to understand the significance of data obtained. With supervision, performs a detailed and comprehensive assessment of a patient presenting with typical and atypical features Performs a detailed and comprehensive assessment of a patient presenting with complex or multiple problems, or in special groups.

NOVICE 2 months BASIC 11 months STAGE 1 Trainee 2 Does not formulate using a BioPsychoSocial model ?Score Trainee 1 Does not formulate using a BioPsychoSocial model ?Score Below standard* for Stage 1 Meets standard* for Stage 1 Above standard* for Stage 1 6. Data synthesis n/a WBA Assessment example – Case-based discussion

WBA Tool – Case-based Discussion (CbD) Occurs in regular supervision time Discussion based on existing case notes Used to assess –Clinical reasoning –Decision making –Integration of medical knowledge with case management –Ability to document these skills Assessor feedback provided to trainee is key

CbD Protocol Trainee arranges time with assessor Trainee selects four cases and gives the files to the assessor Assessor chooses one of the four cases Trainee discusses case with assessor for approx. 15–20 mins Assessor may prompt trainee on further discussion points – guidance on CbD tool Assessor provides feedback to trainee Assessor rates performance (9-point scale) Next learning steps discussed and agreed on Form is signed by assessor and trainee Trainee keeps form and updates learning plan

WBA Tool – Mini-Clinical Evaluation Exercise (Mini-CEX) Assessor observes a trainee in a clinical encounter with a real patient Feedback provided on the agreed clinical task being assessed Ideally, patient is new to trainee, at least in terms of clinical skill being assessed Assessor should be –Familiar with Mini-CEX process –Familiar with the patient

Mini-CEX Used to assess –History taking process –History taking content –Mental state exam –Physical exam skills –Communication skills –Data synthesis –Organisation and efficiency Important: identify assessment focus before commencing exercise

Mini-CEX Protocol Trainee arranges with assessor to observe Mini-CEX Trainee and assessor agree on competencies being assessed Mini-CEX occurs: –Clinical encounter, observed by assessor (15–20 mins) –Discussion and feedback follows immediately (10–15 mins) –Self-reflection time –Assessor rates performance on Mini-CEX form (5–10 mins) Assessor takes no part in encounter unless intervention required for patient safety Trainee and assessor discuss feedback, agree on next steps and sign form Trainee keeps form and updates learning plan

WBA Tool– Observed Clinical Activity (OCA) Similar in format to summative Observed Clinical Interview (OCI) Occurs over two sessions Trainee observed for the duration of an initial patient assessment OCAs are good preparation for summative OCI in Stage 3 OCA and OCIs are different: OCA is formative but OCI is summative Trainees should be encouraged to use the OCA tool from Stage 1 as this will assist them to prepare for the OCIs.

WBA – Observed Clinical Activity (OCA) Examines –History taking and content –Mental state and relevant physical exam skills –Data synthesis –Management plan skills

Observed Clinical Activity (OCA) – Protocol Trainee arranges with assessor to do OCA Trainee nominates a clinical case, ideally new to trainee Full OCA requires two 1-hour sessions OCA (session 1) Clinical encounter50 mins Post-encounter feedback10 mins Thinking/self reflectionTrainee’s own time OCA (session 2) Presentation of Assessment & Viva20 mins Presentation of Plan & Viva20 mins Feedback10 mins Total110 mins

Observed Clinical Activity (OCA) – Protocol continued Feedback is given by assessor Trainee and assessor discuss and agree on next steps, and sign the form Trainee keeps the form and updates the learning plan

WBA Tool – Professional Presentation Used in situations such as –Journal club –Case presentations –Community education presentations –Clinical audits –Grand rounds –In-service presentations, etc. Assesses aspects of typical presentation skills expected from trainees –Introduction –Context –Analysis and critique –Delivery –Questions –Quality of information

Professional Presentation

Professional Presentation - Protocol Trainee selects forum and makes all necessary arrangements Develops presentation Trainee arranges for an assessor to attend Trainee presents presentation of at least 30 mins including questions Assessor observes entire presentation Assessor rates performance Assessor provides feedback to trainee immediately after presentation Next learning steps discussed and agreed on, form signed Trainee retains form in their file

Child & Adolescent WBA Pilot Study Feedback Two barriers to uptake –“Buy-in” –Time But, with more use and familiarity, WBAs should fit into regular supervision time After initial concern, trainees and supervisors found a rating scale useful Trainees found them helpful in –gauging their progress –highlighting areas to work on View that structured process would aid struggling trainees

EPAs – What is an EPA? Entrustable Professional Activity Specific activities that a trainee must –Demonstrate their ability to perform; and –Perform with sufficient independence with distant (reactive) supervision Supervisor must be confident that the trainee –Knows when to ask for additional help –Will seek timely assistance An EPA should be –A task of high importance (core business) –A high-risk or error prone task –A task that is exemplary of a number of CanMEDS roles

EPAs – Stage 1 and 2 General Psychiatry Stage 1 General Psychiatry 1.Producing discharge summaries and organising appropriate transfer of care. 2.Initiating an antipsychotic in a patient known to have schizophrenia. 3.Active participation in the multidisciplinary team meeting. 4.Providing an explanation to a family about a young adult’s major mental illness. Stage 2 General Psychiatry 1.Demonstrating proficiency in all the expected tasks associated with prescription, administration and monitoring of ECT. 2.The application and use of the Mental Health Act. 3.Assessment and management of risk of harm to self and others. 4.The safe and effective use of clozapine in psychiatry. 5.Cultural competence.

EPAs – Standards of Entrustment BASIC PROFICIENT ADVANCED STAGE 1STAGE 2 STAGE 3 PGY 1/2 LOW INDEPENDENCE DEVELOPMENTAL TRAJECTORY HIGH INDEPENDENCE HIGH LEVELS SUPERVISION LOW LEVELS SUPERVISION JUNIOR CONSULTANT

EPAs – Standards of Entrustment As with WBAs, EPAs are entrusted at the following levels –Stage 1 – Basic –Stage 2 – Proficient –Stage 3 – Advanced These standards are described by the Developmental Trajectory which traces the development of a trainee through training Developmental Descriptors help to define the standards

How do the EPA Handbook and RANZCP Certificate of Entrustment work? All EPAs are detailed in the EPA Handbook, which is available on the CBFP website EPA Handbook is helpful for supervisors and trainees The website version is always the most up-to-date Every EPA also has a RANZCP Certificate of Entrustment (COE) –Summary version of full EPA –Sign-off required on RANZCP COE to certify entrustment

Process of Entrusting an EPA Insert full EPA here

Process of Entrusting an EPA

EPA Handbook document contains a full description of the knowledge, skills and attitudes required to gain entrustment of that professional activity May or may not be attained in dedicated supervision time Entrustment decision draws on all data available to supervisor –WBA performance –Observation –Information from other staff/allied health/etc

Entrustment Example: Consultation– Liaison Psychiatry EPA Care for a patient with delirium Skill: Negotiates clinical role throughout the course of the delirium episode Trainee may assess patient, liaise with staff, etc. On the following day, trainee discusses this case with a supervisor in a CbD as Workplace-based Assessment Skill: Considers the patient’s capacity to consent and any implications Trainee may have consented a different patient with a different issue on another occasion and completed a CbD Knowledge: Accesses, appraises and applies best level of evidence. Supervisor has observed trainee working with a number of patients and considers their knowledge to be good.

How WBAs Help Gauge Readiness to Entrust EPAs WBAs provide an evidence base The structured feedback provided by a WBA highlights trainees’ strengths and weaknesses They can measure progress over time WBAs can be used to assess difficult or complex tasks

Process of Entrusting an EPA When the supervisor judges the trainee to be competent to perform the EPA with only distant, reactive supervision → entrustment achieved To document entrustment –Trainee prints the RANZCP COE –Supervisor completes RANZCP COE –RANZCP COE signed by supervisor, trainee and DOT –EPAs attainment will be reported to the College on the In-Training Assessment report at the end of each rotation

Process Of Entrusting An EPA

Questions Any questions? Further resources: –CBFP website – new and updated: –CBFP –Information is also available at the CBFP Booth