Competency Tools to Measure Resident Progress Carol Vandenakker-Albanese, MD Residency Program Director Department of PM&R UC Davis Health System November 15, 2014

ACGME Outcome Project Competency-based education focused on learner performance in reaching specific objectives Long-term initiative by ACGME to emphasize educational outcomes in the accreditation of residency training programs Six core competencies for resident education Medical Knowledge Patient Care Practice Based Learning and Improvement Systems Based Practice Professionalism Interpersonal and Communication Skills

The ‘Next Accreditation System’ in a Nutshell Continuous Accreditation Model – annually updated Based on annual data submitted, other data requested, and program trends RRCs will measure compliance through the evaluation of annual program data elements including: Milestones (submitted twice a year) ACGME Resident Survey ACGME Faculty Survey Procedural /Case Log Data

Procedural Logs: Required (Observed and Performed) EMG/NCS Axial epidural injection Axial facet Peripheral joint/intraarticular injection Tendon sheath/bursa injection Trigger points Peripheral nerve injection Botulinum Toxin Injection Phenol injection Programming baclofen pump Refilling intrathecal baclofen pump Ultrasound extremity Ultrasound guidance

Case Logs: Optional (admission, consult, outpatient) Acute neuromuscular pain Amputation Brain injury Chronic pain management Geriatric rehab Neurologic disorders Oncologic rehab Pediatric rehab Post fracture and post-operative joint arthroplasty Pulmonary and cardiac Rheumatologic Soft Tissue disorders Spinal Cord Injury Stroke

Milestones are based on concept of Entrustable Professional Activity (EPA’s) Real life patient care episodes, usually composed of elements of most if not all “competencies” Benchmark of performance is the ability to be entrusted to perform care with “indirect supervision with direct supervision available” Progression is then the achievement of EPA’s of increasing difficulty, risk, or sophistication Proficiency is then the achievement of the most sophisticated EPA’s required of the resident

Uses and Implications of Milestones ACGME Accreditation –continuous monitoring of programs; lengthening of site visit cycles Public Accountability –report at a national level on competency outcomes Community of practice for evaluation and research, with focus on continuous improvement Residency Programs Guide curriculum development More explicit expectations of residents Support better assessment Enhanced opportunities for early identification of under-performers

Challenges/Opportunities Culture Change and Faculty Development Program Directors, Designated Institutional Officials Faculty Selecting Review Committee Members Improving quality of faculty observations/documentation The “Community of Educators” in each specialty has come together and agree on: core elements of the competencies (Milestones) levels of performance core methods of assessment

Program’s Assessment of Residents Evaluation tools Competency Committee Semi-Annual Reporting to ACGME on each milestone for all residents Annual reporting to ABPMR on resident status and evaluations

Toolbox: Competancy Evaluation Tools Begin by using existing tools and faculty observations Exams: Written, Oral, SAE Clinical Observation: Simulation, ROCA Multi-Source Feedback: 360 evaluations, chart reviews Procedural Skill Evaluation Workshops Clinical assessments Case Logs

Milestone Assessment Determine which tools you will use to assess each milestone Determine which rotations/ faculty are best to evaluate individual milestones Determine which milestones may require additional tools (Yearly updated Resident CV/ self-evaluation)

27 PM&R Milestones Patient Care: 7 Systems Based Practice: 3 Practice-Based Learning and Improvement: 3 Professionalism: 3 Interpersonal and Communication Skills: 2 Medical Knowledge: 9

PC 2. Physiatric Physical Exam (including medical, neurological, and gait adapted for age and impairment) Rotations: Inpatient: General rehab, SCI, TBI Outpatient: MSK, neurorehab Peds Tools: ROCA PE workshop

SBP 2. Team approach to enhance patient care coordination SBP 2. Team approach to enhance patient care coordination. Rehabilitation team members may include OT, PT, SLP, etc. Rotations: Inpatient adult and pediatric rehab Consults Outpatient multidisciplinary clinics Tools: 360 evaluations Faculty evaluations

PROF 3. Professional behaviors and accountability to self, patients, society and the profession Rotations: Any (not all) May want to pick most stressful Tools Faculty evaluations Program director/ coordinator evaluations Review of CV – involvement in service activities

MK 6. Musculoskeletal Disorders. Rotations: Outpatient clinics: sports and spine Tools: Faculty evaluations ROCA Chart reviews SAE/ Written test Mock Oral Exam

Using Milestones as Evaluation Tool Self Assessment on milestones annually Schedule milestone assessments throughout year Incorporate milestones into rotation objectives/ evaluations Incorporate selected milestones into electronic evaluations

Faculty Education Faculty must be familiar with milestones Reminder: Milestone levels NOT equivalent to PGY level Distribute evaluation tools at start of rotation Identify which faculty are appropriate to evaluate which milestones Review/ re-educate faculty based on quality of evaluations

Key Elements of Quality Evaluation Trained Observers- Core Faculty Common understanding of the expectations Sensitive “eye” to key elements Consistent evaluation of a given level of performance Establish Minimum Numbers of Quality Observations (CCC) Interpreter/Synthesizer Experts Clinical Competency Committee (Resident Evaluation Committee)