Maxine A. Papadakis, M.D. UCSF A Guide to the Assessment of Professionalism in Trainees

Disclosures n Nothing to disclose

Objectives Discuss Assessment of unprofessional behavior Determine Consequences for students who have not obtained satisfactory professionalism skills

Professionalism: What is it? NBME work on defining behaviors 8ListofBehaviors.pdf

Year 1 Year 4 Level of Sophistication Add Social Contract Teaching Professionalism Imparting the Cognitive Base Promoting Self-Reflection Capacity to Internalize Professionalism UndergraduatePracticePostgraduate Cruess, Cruess & Steinert, 2010

Performance Observations of Fellow INPATIENT Fellow seems easily distracted on rounds. Fellow has created a climate of tension, and the medical student and resident on the team are intimidated to ask the fellow questions. CLINIC Fellow needs reminders to fulfill her assigned patient care tasks. Fellow leaves clinic before others

What should you do? n How would this be handled at your institution?

Subsequent Evaluation n SUMMARY COMMENTS u She..mastered the basics… u She is very intelligent, and when motivated, she is able to produce exceptional work. n CONSTRUCTIVE COMMENTS u..put in very little effort u Often shirked even basic requirements, including following up on a bone marrow bx… u..personality conflict with one of the nurse practictioners on the service.. u Needs to try to work more effectively with her team and learn to resolve conflict constructively.

Evaluation of Professionalism

Methods to Assess Professionalism n Evaluation forms n Critical incidents n Multisource (360 degree) evaluations n Peer assessment n Evaluation sessions n Patient assessment n P-MEX, OSCEs n NBME Assessment of Professional Behaviors n Physicianship Evaluation Forms Objective vs subjective but 70% pass subjective-easier when people see numbers

Summary of Professionalism Assessment Methods by Richard and Sylvia Cruess

Evaluation Knowledge Behavior Written Tests 4+0 Oral Examinations31 OSCE22 Inter/Gobal Ratings13 Direct Observation14+ Simulations12 Portfolios /Peer Evaluations23

[APB web page]

The NBME’s Assessment of Professional Behaviors Program n Focus: behaviors in GME and FACULTY n Approach: multisource feedback and web –based training of observers n Purpose: to assess the professional behaviors that are essential for safe, effective, and ethical health care n Goal: provide formative feedback that forms the basis for action Copyright© 2010 by the National Board of Medical Examiners® (NBME®). All rights reserved.

Assessment of Professional Behavior Components n A systematically developed instrument to assess observable behaviors n A web-based system to collect, track and collate multisource feedback responses n A source of quantitative and narrative feedback to learners n An educational program to enhance: u Skill as observers u Skill as feedback providers Copyright© 2010 by the National Board of Medical Examiners® (NBME®). All rights reserved.

Sample Score Report Copyright© 2010 by the National Board of Medical Examiners® (NBME®). All rights reserved.

Evaluation Sessions n All instructors complete an evaluation form; 5-point rating scale and comments n All instructors participate in formal evaluation sessions e very 3-4 weeks at all clerkship sites n Clerkship director provides private feedback to each student the next day

Evaluation Sessions n Significantly improved detection of unprofessional behaviors n Professionalism deficiencies 2X more likely to be detected in inpatient rather than outpatient rotations n Comparing standard checklists, written comments and evaluation sessions: Evaluation sessions significantly improved detection of unprofessional behavior in both inpatient and outpatient settings Hemmer et al. Acad Med; 2000: 75:167

Professionalism Mini- Evaluation Exercise Direct Observation and Peer Evaluation Eight evaluators rate professionalism of resident Two faculty Two peer residents Two junior residents Two non-doctor professionals Good reliability & validity for evaluating professionalism Richard and Sylvia Cruess Y Tsugawa et al Medical Education 2009

UCSF Physicianship Evaluation Forms

III. PROFESSIONAL AND PERSONAL ATTRIBUTES 7. PROFESSIONAL ATTRIBUTES AND RESPONSIBILITIES Cannot be relied upon. Attendance and punctuality are erratic. Student’s whereabouts often unknown. Needs prodding frequently. Am concerned over student’s commitment. Needs reminders in the fulfillment of ward responsibilities including patient care. Allows himself/herself to be too peripheral to active team activities and patient care. Can regularly be relied upon in fulfilling responsibilities as a member of the ward team and in the delivery of patients. Exceptionally conscientious. Outstanding in attendance, dependability and punctuality in team activities and patient care responsibilities. Makes an extra effort to be integral team member and assumes high levels of patient care responsibilities. 8. SELF IMPROVEMENT AND ADAPTABILITY Completely unaware of own inadequacies. Refuses to consider or make changes. Resistant or defensive in accepting criticism. Makes those offering suggestions uncomfortable because of lack of receptiveness. Accepts criticism when offered. Makes an effort to change. Does some supplemental as well as required reading. Outstanding in soliciting and receiving criticism with interest and grace. Able to effect change. Self- motivated to expand knowledge. Intellectually aggressive; makes an extra effort to learn patients problems; extensive supplemental reading. 9. RELATIONSHIPS WITH PATIENTS Often insensitive to patient’s feelings, needs and wishes. Lacking capacity for empathy. Sometimes has difficulty establishing rapport with patients or communication with them. Not always comfortable interacting with others. Relates well to most patients and family members. Seems to enjoy patients and family members and the role as a developing physician. Outstanding in putting patients and/or family members at ease and appropriately communicating medical information to them. Relates well with difficult patients. Shows empathy, compassion and respect. 10. INTERPERSONAL RELATIONSHIPS WITH OTHER MEMBERS OF HEALTH CARE TEAM Insensitive to needs, feelings and wishes of health care team members. Poorly integrated into the team. Sometimes has difficulty relating well to health care team members. Relates well to most of the health care team members. Functions well within the team structure on the wards. Outstanding in respecting the feelings needs and wishes of all health care team members. Highly integrated into the team structure.

Physicianship Evaluation System n Years 3-4

Course of Action n Physicianship Evaluation Form u Unmet professional responsibility u Diminished relationship with health care team n Remediation u Determine student’s rationale for the actions u Explore the student’s understanding of why the actions are unprofessional u Refer the student to her mentor u Refer the student to mental health services u Monitor the student’s behavior longitudinally u Forward feeding u Leave of absence

Physicianship Evaluation System n Years 3-4 n Years 1-2 n Institutional n Graded academic consequences

Physicianship Evaluation Form Years 1-2 n Reliability & responsibility n Self-improvement & adaptability n Relationships with students, faculty, staff & patients n Upholding Medical Student Statement of Principles

Institutional Physicianship Evaluation Form n Unmet professional responsibility Communications USMLE deadlines, immunizations Financial aid Social networking n Lack of self-improvement & adaptability Resistant to feedback, arrogant Unaware of inadequacies n Diminished relationships with Administrative faculty & staff

Physicianship Evaluation Forms Forms Students Years Years TOTAL

Physicianship Form Distribution in 59 students Clerkship/Course # of StudentsMale Female Ob-Gyn Psychiatry Pediatrics Medicine Medicine Subinternship Neurology Surgery Family Medicine Other TOTAL

Outcome of 12 students with 2-3 Physicianship Forms  Mentioned in dean’s letter2  Program notification 1  Withdrawal1  Dismissal 3  Leave of absence2

Unprofessional Behavior in Faculty and Residents

Evaluation of Faculty by Student I felt that her brusque and often rude manner was inappropriate. She came across as belittling and rude to students, but not to patients. Other times, she was kind. It was an emotional rollercoaster. I must stress that she is fiercely intelligent…but she is emotionally labile.

Professionalism as Competence n Defining specific objectives, competencies n Identifying where within the curriculum u they are delivered u they are measured

Rewarding Professionalism n Arnold P. Gold Foundation u Humanism Honor Society Peer nomination n Medical School Performance Evaluation (MSPE) n Faculty rewarding from Respect data

Jordan Cohen 1999 We can measure whether or not our students ‘know’ about professionalism, i.e., whether they can define the word, list its components, recite its historical development, and recognize the contemporary threats to its survival. But how can we know whether they are professionals, i.e., whether they are destined—or at the very least, are likely—as physicians to behave in accordance with professionalism’s demanding tenets? Knowledge, while essential, is not sufficient. Performance is the key.