Introduction to Evaluation Part Three: Evaluating Learners Objectively

Learning Objectives By the end of this session, participants will be able to: Evaluate clinical learners objectively Provide meaningful comments for evaluation Note: This is part three of a three part series. Part one provides an overview of feedback and evaluation and part two covers how to give effective feedback.

Recall from Part One: Evaluation A measurement judging performance given at the completion of a learning experience Comparison with other learners’ performances Often mistaken for feedback Similar to scoring in gymnastics or ice skating

Formative vs. Summative Evaluations Similar to feedback, but at the end of an experience Informs the learner about him/herself (internal) Informs learner of progress Advises learner of observed learning needs Guides self-reflection and motivates improvement Summative A final assessment of overall performance Informs others about the learner (external) A measure of outcome May be used verbatim in medical student performance evaluation (MSPE)/ “Dean’s letter”

Making Evaluations Objective To be meaningful, evaluations need to have context Level and experience of learner What are the expectations (learning objectives) Situations of the environment You should be provided with the learning objectives that offer this context

Example Evaluation Form: Providing Context CLINICAL SKILLS The medical student demonstrated the clinical skills necessary, such as performing patient interviews and/or patient histories, to participate in clinical experiences during preceptorship.   Does Not Meet Expectations Meets Some Expectations Meets Expectations Exceeds Expectations Expectations: Ability to perform a history and physical exam; present in a clear and concise way; garner appropriate clinical information from a variety of sources; document daily records appropriately; perform routine procedures; adapt to a variety of clinical environments Performance/ behavior that does not meet expected levels, performance/ behavior that may be poor and is significantly less than expected levels. (i.e. lateness, inappropriate attire or language, or difficulty with communication) behavior that meets some expectations but not all of the criteria behavior that meets expectations behavior that significantly exceeds expectations for a first year medical student. For example, students who exhibit abilities, performance, qualities, values and characteristics that go above and beyond the standard expectations References to give context to your evaluation “Based on the statement and definition below for a first year medical student. Please refer to your Preceptor Handbook and course objectives for additional information.”

Example Evaluation Form: Context Resident Competencies & Milestones Competency Domain: Patient Care Competency: Gather essential and accurate information about the patient Developmental Milestones: Level 1 Level 2 Level 3 Level 4 Level 5 Either gathers too little information or exhaustively gathers information Linkage of signs and symptoms with prior experiences Some pertinent positives and negatives Uses broad diagnostic categories Pattern-recognition Real-time development of a differential Precise diagnoses for common problems reached with ease and efficiency Still limited on more complex problems Unconscious gathering of essential and accurate information Targeted and efficient with most clinical problems In this example, the domain is Patient Care, the Competency is “Gather essential and accurate information about the patient.” The resident should be observed taking a history to see if he is able to demonstrate these skills.

Tools: The Evaluation Form Effective tools help make an evaluation objective Good evaluation forms have “anchors” Anchors allow for measurement of observable behaviors: Level 1 Level 2 Level 3 Level 4 Level 5 Either gathers too little information or exhaustively gathers information Linkage of signs and symptoms with prior experiences Some pertinent positives and negatives Uses broad diagnostic categories Pattern-recognition Real-time development of a differential Precise diagnoses for common problems reached with ease and efficiency Still limited on more complex problems Unconscious gathering of essential and accurate information Targeted and efficient with most clinical problems anchors

Tools: Use the Timeline Review the evaluation form with the learner at the start of a rotation Relating this form to learning objectives is helpful Keep notes of feedback given to the learner; refer to these notes while completing the evaluation form Orientation Feedback Feedback Evaluation

For Written Comments Use objectives and/or competencies for the basis of each comment Use specific examples Avoid generic “feel good” comments

Tools: Relating to Learning Objectives & Competencies Neurological exam (patient care) Teaches the team/brings articles (practice-based learning) Receptive to/incorporates feedback (professionalism) Interrupts other team members (interpersonal skills) Overconfident (patient care, knowledge) Texts during rounds (professionalism) Use of resources (systems-based practice)

Tools: Specific Examples “Well organized patient presentation of a complicated patient with _____.” “Was able to apply the principles of acid-base physiology.” “Demonstrated proper use of sterile technique.” “Thoroughly documented the history and physical, but needs to increase the breadth of the differential diagnosis and expand the assessment to better demonstrate their clinical problem solving abilities.”

Exercise: Practice Writing Formative & Summative Comments Using the phrases below, write a few evaluative sentences: one set formative (directed to the student) and one set summative (appropriate for a final evaluation). “Brought articles” “Impressed by his DDx” “Needs to read more about renal Pathophysiology” “Often late for rounds” “Thorough history and physical examinations” “Will make a great house officer” Formative comments Summative comments

Examples: Formative & Summative Comments Notice we left out the “will be a good house officer” which does not describe this learner’s performance. This is a judgment call: if the student was given feedback and did not improve, could include in summative comments. Formative comment Summative comment “Dilbert, your DDx were very good and related to the patient’s history and physical examination. You need to plan ahead to make sure you are on time for rounds. You need to learn more about renal pathophysiology as this was identified as an area of weakness on rounds.” “Dilbert’s overall knowledge base is good, his H&Ps are thorough, and his DDX were well thought out and appropriate. He made important contributions to the team by researching articles helpful to the care of complicated patients.”

Questions? Contact: Jennifer Hagen, MD Office of Faculty Development jmhagen@med.unr.edu April Heiselt, PhD Office for Community Faculty aheiselt@med.unr.edu