Evaluation of Learners Effective Techniques and Strategies Rationale: Your primary focus is on learning to be good doctors. However, another expectation of you is to be teachers of students. For you to do so effectively, you need to understand and use elements improtant to good teaching, oen fo which si evalaution. Evalaution can serve different purposes depending on the audience. Jennie Hsu-Lumetta, M.D. March 16, 2005 9/19/2018

“Clinical Learning is Experience Examined…” Ende and Davidoff, 1992 9/19/2018

Goals and Objectives 1. Define the purpose of evaluations. 2. Identify the steps to providing effective evaluations. 3. List some potential barriers/errors to providing effective evaluation . Items 2-5 refer to the elements of effective evaluation 9/19/2018

What Will Be Covered. . . 1. Definition, Purposes, Types of Evaluation 2. Pre-Evaluation Activities 3. Completing the Evaluation 4. Writing Good Comments 5. Discussing Evaluation 6. RIME Model for Clinical Evaluation Items 2-5 refer to the elements of effective evaluation 9/19/2018

Definition of Evaluation To determine the value or worth of a Program, Product, Performance Process I like this definition. Note that the key element is a determination of value or worth. We seek your expert opinion of the student’s performance. 9/19/2018

Evaluation of Learners The process by which the teacher assesses the learners’ knowledge, skills, and attitudes, based on criteria related to educational goals. I like this definition. Note that the key element is a determination of value or worth. We seek your expert opinion of the student’s performance. 9/19/2018

Why Evaluate? Past: Assess performance Present: Improve performance Future: Identify future needs Your ability to function as an evaluator depends greatly on having a clear understanding of why you are doing the evalaution in the first place. I like to think about evaluation in terms of how it is used in time. 9/19/2018

Uses of Medical Student Evaluations Use Purpose Audience Past Assessment Clerkship Director Present Improvement Student Future Dean’s Letter Dean When we apply this concept of temporal orientation to medicine, the uses, purposes and audiences are pretty clear. While the evalaution you complete on the students can serve all three purposes, the past orientation of student assessment is the primary purpose of the formal evalaution form. 9/19/2018

Uses of Resident Evaluations Use Purpose Audience Past Assessment Program Director Present Improvement Resident Future Program Fellowship, Director’s Letter Employers When we apply this concept of temporal orientation to medicine, the uses, purposes and audiences are pretty clear. While the evalaution you complete on the students can serve all three purposes, the past orientation of student assessment is the primary purpose of the formal evalaution form. 9/19/2018

What to Evaluate? Attitude Skills Knowledge 9/19/2018

ASK’s Attitude, Skills, Knowledge Competency includes the following: Attitude: (as observed via behaviors) Prepared/Punctuality/Proactive Follow-up with patient/educational issues Skills – context specific Knowledge Has the information Able to analyze and synthesize information Able to apply it effectively and efficiently 9/19/2018

ACGME Competencies 1. Patient Care 2. Medical Knowledge 3. Practice-Based Learning and Improvement 4. Interpersonal and Communication Skills 5. Professionalism 6. Systems-Based Practice 9/19/2018

ACGME Competencies Patient Care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health Medical Knowledge about established and evolving biomedical, clinical, and cognate (e.g. epidemiological and social-behavioral) sciences and the application of this knowledge to patient care 9/19/2018

ACGME Competencies Practice-Based Learning and Improvement that involves investigation and evaluation of their own patient care, appraisal and assimilation of scientific evidence, and improvements in patient care Interpersonal and Communication Skills that result in effective information exchange and teaming with patients, their families, and other health professionals 9/19/2018

ACGME Competencies Professionalism, as manifested through a commitment to carrying out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population Systems-Based Practice, as manifested by actions that demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value 9/19/2018

ACGME - Evaluation of Residents use of dependable measures to assess 6 areas of competency mechanisms for providing regular and timely performance feedback to residents a process involving use of assessment results to achieve progressive improvements in residents' competence and performance 9/19/2018

Types of Evaluation Formative Evaluation Summative Evaluation 9/19/2018

Types of Evaluation Formative Evaluation Assess a learner’s progress toward educational goals Throughout the teaching process Basis for feedback for the learners and to make teaching adjustments as needed to learner to reach learning goals 9/19/2018

Types of Evaluation Summative Evaluation Final assessment of a learner End of teaching/learning experience Used to judge a learner’s mastery of the educational goals 9/19/2018

How to Evaluate? Assessment/Evaluation Feedback Evaluation Formative (ongoing) Feedback Evaluation Summative (end of rotation) 9/19/2018

Elements of Effective Assessment Evaluation Prerequisites Making Good Comments Aligning Comments with Scaled Questions Discussing the Evaluation with the Learner Source: David Rubin, MD and James Woodruff, MD. Teaching Skills for the Medical Educator Workshop. University of Chicago, March 23, 2002 For the evaluation to be most effective for any of the three major purposes, it is important for you to keep a few basic principles in mind. We will take a few minutes to discuss each one. 9/19/2018

Effective Assessment 1: Evaluation Prerequisites Understand expectations of the learner Understand the evaluation criteria Understand the survey instrument Have appropriate expectations Provide on-going feedback Provide opportunities for observation Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties: PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner. Describe anchor points and evaluation criteria using current student evaluation form. 9/19/2018

Key components of Evaluation Observation of the Learners Questioning Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties: PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner. Describe anchor points and evaluation criteria using current student evaluation form. 9/19/2018

Key components of Evaluation : Observation of the Learners The teacher can observe learner performance for direct evaluation of knowledge, skills, and attitudes. Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties: PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner. Describe anchor points and evaluation criteria using current student evaluation form. 9/19/2018

Key components of Evaluation : The One minute Observation Explain the purpose of the observation to the learner Explain how the observation will occur Select one skill for observation Inform the patient of what will take place Observe for a brief period without interrupting Leave the room and give immediate feedback Use the information gain to plan your teaching Repeat to observe learner’s skills over time Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties: PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner. Describe anchor points and evaluation criteria using current student evaluation form. 9/19/2018

Key components of Evaluation : Questioning of the Learners Types of Questions Levels of Questions Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties: PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner. Describe anchor points and evaluation criteria using current student evaluation form. 9/19/2018

Key components of Evaluation : Types of Questions Open-ended What? How? Why? Closed-ended Do? Can? Have? Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties: PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner. Describe anchor points and evaluation criteria using current student evaluation form. 9/19/2018

Key components of Evaluation : Level of Questions Recall Analysis/Synthesis Application Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties: PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner. Describe anchor points and evaluation criteria using current student evaluation form. 9/19/2018

Effective Assessment 2: Completing the Form Write narrative comments first Comments are most important!! Grades are a “blunt instrument” Dr. Rubin, U of Chicago Dept. of Medicine Give specific examples Keep comments to focused behaviors Connect comments to ratings Check for internal consistency Activity 2: Peer Evaluation. 9/19/2018

Sources of Comments Direct observation of learner Written records Responses to questions Homework assignments Interactions with healthcare team, patients Learner self-evaluation Feedback/Midpoint evaluation Interventions with the learner Learner’s response to interventions 9/19/2018

Effective Assessment 3: Writing Effective Comments Strengths and weaknesses Areas of concern Suggestions for improvement Ensure internal consistency Focus on behaviors not personalities There are different ways to organize your comments. On our student form, the gernal summary allows you to write strengths and weaknesses and areas of coner. There is also a section forimprovement in the student feedback section. Optional Activity 3: Writing Good Comments Here are sample comments written about students by residents and interns Please read the comments and discuss with your team member in terms of:   1.      The abilities of the student 2.      Level of specificity 3.      Utility for: a.       The student’s continued learning b.      The clerkship director’s final grade c.       Material for future letter of recommendation How might the comments be more useful? 9/19/2018

Effective Assessment 4: Discuss with Learner Set a time Allow enough time for discussion Have the learner perform a self-assessment Explain your rationale Emphasize changes in behavior and performance that leads to desired competency Suggest areas for future improvement Give evaluation in oral and written forms *Adding feedback element increases the usefulness of the evaluation and avoids surprises Closes the loop and addresses present deficits and future needs as well as past eprformance. 9/19/2018

Avoid rating errors: The “halo/horn” effect: Restriction range: based on reports of prior performance Restriction range: using the same rating for all components of the evaluation Rating nonperformance attributes: using behavior attributes when rating nonbehavioral qualities Evaluate performance ONLY 9/19/2018

RIME Model for Clinical Evaluation Developed by Dr. Louis Pangaro Model for student evaluation that was credible and reliable Student performance -> Progression of 4 developmental steps Research shows that it is predictive of future performance 9/19/2018

RIME Model for Clinical Evaluation 4 Stages of Clinical Development: Reporter Interpreter Manager Educator 9/19/2018

RIME Model for Clinical Evaluation Reporter collect and report data (by the M3 year) Interpreter interpret history, PE findings and lab results, create & prioritize a problem list and differential diagnosis (by end of M3 year) Manager select the most appropriate diagnostic and treatment options (by 2nd & 3rd residents) Educator identify and address knowledge gaps (senior residents) 9/19/2018

Proficient Reporter Interpreter Interview & History Physical examinations Demonstrates adequate medical knowledge Generates adequate problem list Demonstrates respect for patients Consistently reliable & honest Interpreter Interprets basic test data Integrates medical knowledge with findings Generates several reasonable explanations Can discuss issues & process of informed consent 9/19/2018

Advanced Manager Develop diagnostic & therapeutic plans Makes decisions using benefit/risk analysis Uses evidence-based approach Performs as a reliable member of the team Manages relationships with patients 9/19/2018

Outstanding Educator Demonstrates leadership Demonstrates self-directed learning Exhibits critical reading skills Demonstrates teaching skills 9/19/2018

Summary for Effective Evaluators: Understand learner expectations Understand the evaluation instrument: E-value, RIME model Include thoughtful comments Connect comments to rating scales Have multiple opportunities for assessing performance Evaluate performance ONLY Discuss evaluation with the learner 9/19/2018

Special thanks: Nancy F. Barrett Carol Packard 9/19/2018

Case You sit down with your student on her last day of the rotation although you’ve never been comfortable giving evaluations. Knowing evaluations are supposed to be balanced, you try to list several positive things and then list problems. When you mention that the student’s cardiac patient who spent an extra day in the hospital because she didn’t track down a stress test result in time on Friday before the cardiology department closed, she angrily bursts out how difficult it is on a team where no one will give her any help. She also says that you haven’t spent as much time reviewing write-ups with her as other residents. 9/19/2018