1. Evaluating Learners Jennifer L. Middleton, MD, MPH, FAAFP
Academic Medicine Rotation
August 2014
This is a very superficial overview of this topic – you can get master’s degrees in this! But it will give you the basics to help increase the quality of the evaluations you provide.

2. Objectives Describe possible goals of evaluating medical learners.
List 3 challenges to accurate medical learner evaluation.
Define “anchor point.”

3. What is evaluation? Why do we do it? Accreditation (ACGME, AAMC)
Assess competence
Gauge academic progress
Predict performance
Guide improvement
Assign grades
Judge program effectiveness

4. What’s the difference between feedback and evaluation?

5. Feedback
Evaluation
Provides information about performance
YES
Includes judgment about quality NO
Example
Your non-verbal skills showed your patient that you were interested in her story.
Your doctor-patient communication skills are appropriate for your level of training.

6. Summative evaluation
vs.
Formative evaluation
at the end
during

7. How is formative evaluation different than feedback?
provides assessment of quality
How is formative evaluation different than feedback?

8. what an individual does
Process performance
vs.
Outcome performance
what an individual does
the result
Schmutz et al. Five Steps to Develop Checklists for Evaluating Clinical Performance: An Integrative Approach. Acad Med. 2014;89:

9. Subjective vs. objective measures
Process performance:
Subjective vs. objective measures
based on predefined scoring categories using specific criteria
based on clinical expertise of the rater
subjective = problems with interrater reliability
objective = done well should eliminate that problem
Schmutz et al. Five Steps to Develop Checklists for Evaluating Clinical Performance: An Integrative Approach. Acad Med. 2014;89:

10. Objective criteria = checklist
Is one of these better? (define “better”!)

11. Anchor points

12. Evaluation challenges
Grade inflation
Halo effect
Vague descriptions
How might objective evaluation measures help with these?
(does it completely?)

13. One possible process for completing an evaluation:
Complete yes/no and/or Likert scale questions first.
Expand on any above average or below average scores.
Offer specifics re: taking above average skills to the next level.
Offer specifics re: improving below average marks.
Provide your best estimate of the learner’s capability to improve below average areas.

14. Creating evaluation forms
Summative or formative?
Process or outcome?
Assess objectives
Legible – plenty of room!

15. Types of evaluation questions
Yes/No
Likert scale
Qualitative
What questions would work best to…
Determine if your objectives were met?
Rate skill at a procedure/behavior?
Elicit ideas you might not have thought of?

16. The 6(7) ACGME (AOA) Competencies
Medical knowledge
Patient Care
Practice Based Learning and Improvement
Systems Based Practice
Interpersonal Skills and Communication
Professionalism
Osteopathic Principles and Practices