1. Evaluation of Learners
Effective Techniques and Strategies
Rationale: Your primary focus is on learning to be good doctors. However, another expectation of you is to be teachers of students. For you to do so effectively, you need to understand and use elements improtant to good teaching, oen fo which si evalaution. Evalaution can serve different purposes depending on the audience.
Jennie Hsu-Lumetta, M.D.
March 16, 2005
9/19/2018

2. “Clinical Learning is Experience Examined…”
Ende and Davidoff, 1992
9/19/2018

3. Goals and Objectives 1. Define the purpose of evaluations.
2. Identify the steps to providing effective evaluations.
3. List some potential barriers/errors to providing effective evaluation .
Items 2-5 refer to the elements of effective evaluation
9/19/2018

4. What Will Be Covered. . . 1. Definition, Purposes, Types of Evaluation
2. Pre-Evaluation Activities
3. Completing the Evaluation
4. Writing Good Comments
5. Discussing Evaluation
6. RIME Model for Clinical Evaluation
Items 2-5 refer to the elements of effective evaluation
9/19/2018

5. Definition of Evaluation
To determine the value or worth of a
Program,
Product,
Performance
Process
I like this definition. Note that the key element is a determination of value or worth. We seek your expert opinion of the student’s performance.
9/19/2018

6. Evaluation of Learners
The process by which the teacher assesses the learners’
knowledge, skills, and attitudes, based on criteria related to educational goals.
I like this definition. Note that the key element is a determination of value or worth. We seek your expert opinion of the student’s performance.
9/19/2018

7. Why Evaluate? Past: Assess performance Present: Improve performance
Future: Identify future needs
Your ability to function as an evaluator depends greatly on having a clear understanding of why you are doing the evalaution in the first place. I like to think about evaluation in terms of how it is used in time.
9/19/2018

8. Uses of Medical Student Evaluations
Use Purpose Audience
Past Assessment Clerkship
Director
Present Improvement Student
Future Dean’s Letter Dean
When we apply this concept of temporal orientation to medicine, the uses, purposes and audiences are pretty clear. While the evalaution you complete on the students can serve all three purposes, the past orientation of student assessment is the primary purpose of the formal evalaution form.
9/19/2018

9. Uses of Resident Evaluations
Use Purpose Audience
Past Assessment Program
Director
Present Improvement Resident
Future Program Fellowship, Director’s Letter Employers
When we apply this concept of temporal orientation to medicine, the uses, purposes and audiences are pretty clear. While the evalaution you complete on the students can serve all three purposes, the past orientation of student assessment is the primary purpose of the formal evalaution form.
9/19/2018

10. What to Evaluate?
Attitude
Skills
Knowledge
9/19/2018

11. ASK’s Attitude, Skills, Knowledge
Competency includes the following:
Attitude: (as observed via behaviors) Prepared/Punctuality/Proactive
Follow-up with patient/educational issues
Skills – context specific
Knowledge
Has the information
Able to analyze and synthesize information
Able to apply it effectively and efficiently
9/19/2018

12. ACGME Competencies 1. Patient Care 2. Medical Knowledge
3. Practice-Based Learning and Improvement
4. Interpersonal and Communication Skills
5. Professionalism
6. Systems-Based Practice
9/19/2018

13. ACGME Competencies
Patient Care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health
Medical Knowledge about established and evolving biomedical, clinical, and cognate (e.g. epidemiological and social-behavioral) sciences and the application of this knowledge to patient care
9/19/2018

14. ACGME Competencies
Practice-Based Learning and Improvement that involves investigation and evaluation of their own patient care, appraisal and assimilation of scientific evidence, and improvements in patient care
Interpersonal and Communication Skills that result in effective information exchange and teaming with patients, their families, and other health professionals
9/19/2018

15. ACGME Competencies
Professionalism, as manifested through a commitment to carrying out professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population
Systems-Based Practice, as manifested by actions that demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value
9/19/2018

16. ACGME - Evaluation of Residents
use of dependable measures to assess 6 areas of competency
mechanisms for providing regular and timely performance feedback to residents
a process involving use of assessment results to achieve progressive improvements in residents' competence and performance
9/19/2018

17. Types of Evaluation Formative Evaluation Summative Evaluation
9/19/2018

18. Types of Evaluation Formative Evaluation
Assess a learner’s progress toward educational goals
Throughout the teaching process
Basis for feedback for the learners and to make teaching adjustments as needed to learner to reach learning goals
9/19/2018

19. Types of Evaluation Summative Evaluation Final assessment of a learner
End of teaching/learning experience
Used to judge a learner’s mastery of the educational goals
9/19/2018

20. How to Evaluate? Assessment/Evaluation Feedback Evaluation
Formative (ongoing)
Feedback
Evaluation
Summative (end of rotation)
9/19/2018

21. Elements of Effective Assessment
Evaluation Prerequisites
Making Good Comments
Aligning Comments with Scaled Questions
Discussing the Evaluation with the Learner
Source: David Rubin, MD and James Woodruff, MD. Teaching Skills for the Medical Educator Workshop. University of Chicago, March 23, 2002
For the evaluation to be most effective for any of the three major purposes, it is important for you to keep a few basic principles in mind. We will take a few minutes to discuss each one.
9/19/2018

22. Effective Assessment 1: Evaluation Prerequisites
Understand expectations of the learner
Understand the evaluation criteria
Understand the survey instrument
Have appropriate expectations
Provide on-going feedback
Provide opportunities for observation
Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties:
PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner.
Describe anchor points and evaluation criteria using current student evaluation form.
9/19/2018

23. Key components of Evaluation
Observation of the Learners
Questioning
Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties:
PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner.
Describe anchor points and evaluation criteria using current student evaluation form.
9/19/2018

24. Key components of Evaluation : Observation of the Learners
The teacher can observe learner performance for direct evaluation of knowledge, skills, and attitudes.
Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties:
PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner.
Describe anchor points and evaluation criteria using current student evaluation form.
9/19/2018

25. Key components of Evaluation : The One minute Observation
Explain the purpose of the observation to the learner
Explain how the observation will occur
Select one skill for observation
Inform the patient of what will take place
Observe for a brief period without interrupting
Leave the room and give immediate feedback
Use the information gain to plan your teaching
Repeat to observe learner’s skills over time
Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties:
PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner.
Describe anchor points and evaluation criteria using current student evaluation form.
9/19/2018

26. Key components of Evaluation : Questioning of the Learners
Types of Questions
Levels of Questions
Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties:
PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner.
Describe anchor points and evaluation criteria using current student evaluation form.
9/19/2018

27. Key components of Evaluation : Types of Questions
Open-ended
What?
How?
Why?
Closed-ended
Do?
Can?
Have?
Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties:
PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner.
Describe anchor points and evaluation criteria using current student evaluation form.
9/19/2018

28. Key components of Evaluation : Level of Questions
Recall
Analysis/Synthesis
Application
Before you can evalaute a studetn or peer, Drs. Rubin and Woodruff suggest certains thags as prerequisties:
PASS out current student evalaution form. Ask about experiences using the form. Emphasize improtance of narrative comments. Emphasize improtance of completing forms in a tiemly manner.
Describe anchor points and evaluation criteria using current student evaluation form.
9/19/2018

29. Effective Assessment 2: Completing the Form
Write narrative comments first
Comments are most important!!
Grades are a “blunt instrument”
Dr. Rubin, U of Chicago Dept. of Medicine
Give specific examples
Keep comments to focused behaviors
Connect comments to ratings
Check for internal consistency
Activity 2: Peer Evaluation.
9/19/2018

30. Sources of Comments Direct observation of learner Written records
Responses to questions
Homework assignments
Interactions with healthcare team, patients
Learner self-evaluation
Feedback/Midpoint evaluation
Interventions with the learner
Learner’s response to interventions
9/19/2018

31. Effective Assessment 3: Writing Effective Comments
Strengths and weaknesses
Areas of concern
Suggestions for improvement
Ensure internal consistency
Focus on behaviors not personalities
There are different ways to organize your comments. On our student form, the gernal summary allows you to write strengths and weaknesses and areas of coner. There is also a section forimprovement in the student feedback section.
Optional Activity 3: Writing Good Comments
Here are sample comments written about students by residents and interns
Please read the comments and discuss with your team member in terms of:
1.      The abilities of the student
2.      Level of specificity
3.      Utility for:
a.       The student’s continued learning
b.      The clerkship director’s final grade
c.       Material for future letter of recommendation
How might the comments be more useful?
9/19/2018

32. Effective Assessment 4: Discuss with Learner
Set a time
Allow enough time for discussion
Have the learner perform a self-assessment
Explain your rationale
Emphasize changes in behavior and performance that leads to desired competency
Suggest areas for future improvement
Give evaluation in oral and written forms
*Adding feedback element increases the
usefulness of the evaluation and avoids surprises
Closes the loop and addresses present deficits and future needs as well as past eprformance.
9/19/2018

33. Avoid rating errors: The “halo/horn” effect: Restriction range:
based on reports of prior performance
Restriction range:
using the same rating for all components of the evaluation
Rating nonperformance attributes:
using behavior attributes when rating nonbehavioral qualities
Evaluate performance ONLY
9/19/2018

34. RIME Model for Clinical Evaluation
Developed by Dr. Louis Pangaro
Model for student evaluation that was credible and reliable
Student performance -> Progression of 4 developmental steps
Research shows that it is predictive of future performance
9/19/2018

35. RIME Model for Clinical Evaluation
4 Stages of Clinical Development:
Reporter
Interpreter
Manager
Educator
9/19/2018

36. RIME Model for Clinical Evaluation
Reporter
collect and report data (by the M3 year)
Interpreter
interpret history, PE findings and lab results, create & prioritize a problem list and differential diagnosis (by end of M3 year)
Manager
select the most appropriate diagnostic and treatment options (by 2nd & 3rd residents)
Educator
identify and address knowledge gaps (senior residents)
9/19/2018

37. Proficient Reporter Interpreter Interview & History
Physical examinations
Demonstrates adequate medical knowledge
Generates adequate problem list
Demonstrates respect for patients
Consistently reliable & honest
Interpreter
Interprets basic test data
Integrates medical knowledge with findings
Generates several reasonable explanations
Can discuss issues & process of informed consent
9/19/2018

38. Advanced Manager Develop diagnostic & therapeutic plans
Makes decisions using benefit/risk analysis
Uses evidence-based approach
Performs as a reliable member of the team
Manages relationships with patients
9/19/2018

39. Outstanding Educator Demonstrates leadership
Demonstrates self-directed learning
Exhibits critical reading skills
Demonstrates teaching skills
9/19/2018

40. Summary for Effective Evaluators:
Understand learner expectations
Understand the evaluation instrument:
E-value, RIME model
Include thoughtful comments
Connect comments to rating scales
Have multiple opportunities for assessing performance
Evaluate performance ONLY
Discuss evaluation with the learner
9/19/2018

41. Special thanks:
Nancy F. Barrett
Carol Packard
9/19/2018

42. Case
You sit down with your student on her last day of the rotation although you’ve never been comfortable giving evaluations. Knowing evaluations are supposed to be balanced, you try to list several positive things and then list problems. When you mention that the student’s cardiac patient who spent an extra day in the hospital because she didn’t track down a stress test result in time on Friday before the cardiology department closed, she angrily bursts out how difficult it is on a team where no one will give her any help. She also says that you haven’t spent as much time reviewing write-ups with her as other residents.
9/19/2018