1. The Challenge and Importance of Evaluating Residents and Fellows Debra Weinstein, M.D. PHS GME Coordinators Retreat March 25, 2011

2. What keeps us up at night? Recruiting top candidates Protecting time for education Duty hours compliance Evaluating residents; providing feedback; dealing with poor performance

3. Why is this so important? Duty to trainees –promote professional development –help each reach his/her greatest potential Duty to society –ensure minimum standards of competence –promote development of accurate self- assessment, knowledge of individual limits

4. How do we know theres a gap? Internal reviews ACGME resident survey ACGME citations Individual cases

5. Where are we falling short? Many assigned evaluations are never done Quality of evaluations is variable –Important questions not asked –Only ratings / no commentary –Intentional avoidance of critique Evaluations not consistently reviewed with trainees 360-degree evaluations not implemented

6. Goals Protect current and future patients –Ensure appropriate balance of supervision and independence during training –Provide accurate references –Develop physician skills in self-assessment, understanding individual limits Optimize professional development and satisfaction of individual trainees –Focus on exceptional talent and potential as well as weaknesses –Utilize performance reviews to individualize education and career counseling

7. Trainee assessment must be able to identify the spectrum of problems encountered Insufficient academic progress: knowledge, judgment, decision making Insufficient technical skill Poor professionalism: dishonest, irresponsible, poor work ethic, inappropriate behavior Interpersonal conflicts Substance abuse Mental health issues Learning disabilities; ADD Other: poor organizational skills, carelessness, lack of productivity, etc.

8. PHS Evaluation/Feedback Task Force: Key Recommendations Stronger and more specific PHS policy Steering Committee to oversee and champion the effort Dedicated champion to implement and assess impact Faculty development, resident/fellow training, improved infrastructure Incentives/mandates for faculty

9. Faculty Development Initiatives Institution ABIM Hospital Visit Program, May 2011 Department Train the trainer sessions Grand rounds, retreats, new hire orientation Individual 1:1 coaching, role play

10. Resident/Fellow Training shift in culture: seek, accept and utilized feedback –Formative, not punitive! peer evaluation self-evaluation

11. OSCE (Objective Structured Clinical Exam) Objective: 1) Teaching 2) ?Early identification at risk trainees Focus: Communication/interpersonal skills; patient safety Plan: 100 incoming intern participates during 2011 orientation program Evaluate cost, feasibility, outcomes in considering future utilization

12. Targets for improved infrastructure Templates for 360-degree evaluations, can be customized by program Automated compliance reports from web- based evaluation system Electronic vehicle to immediately communicate a concern

13. How will we measure progress? Pre/post survey of faculty and trainees –knowledge, attitudes, confidence Pre/post comparison of evaluation completion rates, and quality Internal reviews and accreditation reports Discussion with program directors, coordinators, trainees and faculty

14. Key Elements of An Improved Assessment System Sufficient contact between trainees and faculty to allow for assessment Consistent completion of high quality evaluations Effective feedback to trainees Early identification of performance problems with prompt intervention Accessible resources for remediation

15. Coordinators Role: Work with Program Director to define this! May include: Address the need for faculty contact/continuity in scheduling Monitor evaluation completion rates; address non-compliance Help to design and implement plan for obtaining 360-degree evaluations Utilize and customize evaluation templates Help to ensure that residents understand the programs system for assessment: how is this communicated?