1. Medical student evaluations:
How much is Too Much?
L.M. Mellor, MD
Assistant Professor
Rutgers, RWJMS Centrastate Family Medicine Residency Program

2. Medical Student Evaluation Throughout History
Apprenticeships- direct observation and feedback, then out and independent
Schools developed- written exams, some observation/ feed back
Then schools relied more on written exams, little feedback
Now schools are trying many different methods of evaluation in an attempt to produce better educated physicians with more standardized evaluations

3. Current Status of Evaluation of Clinical Competence
Use as many evaluation methods as possible to try to get a complete picture of the student’s abilities.
But, How much is Too Much?
Is this really the best method?

4. For example:
Studies show clerkship grading varies dramatically! (1,2) variability in the number of HONORS- 2-93%!

5. Less than 1% fail a required clerkship! (3)
Over 40% of clerkship directors admitted they had passed a student who should have failed (3)
Giving a student a “Pass” is the kiss of death in a clerkship so…
Grade inflation is a common issue

6. Why Don’t they Fail?--or even get a “Pass”?
Docs don’t want to deal with angry, upset or litigious students (3)
Often there is no set method of dealing with struggling students (4)
Remediation is TOUGH

7. To Make Matters even Worse: Non-cognitive predictors of grades exist
“One study showed that lower clerkship grades were associated with nonwhite race, male gender, older age, lower quality of clerkship experience, and being less assertive and more reticent” (5)
There is a vague sense from most students that the whole grading process is pretty subjective.

8. Many believe there is a solution-
USE AS MANY DIFFERENT “OBJECTIVE” METHODS AS POSSIBLE TO ASSESSS STUDENTS ON CLINICAL ROTATIONS!

9. As an Example-Evaluation of “competence” in FM Rotation
Written standardized exam (the Shelf) OSCE Mini-CEX Logs of pt encounters/see at least 6 pts per day ACGME 6 core competencies for mid-term RIME for Final Evaluation with written evaluation Direct observation with verbal feedback throughout rotation Evaluation of Presentation at clinical off-site location

10. Standardized exams- “SHELF”
Value of these exams to assess knowledge base has been established over years of study
Some students do not perform as well on these exams as they do in clinical situations
These exams are limited in their ability to assess critical thinking skills
These exams cannot accurately assess interpersonal skills

11. Direct observation with patient and written/verbal feedback
Personalized, more reliable than would be expected (study compared stud results to res directors(6)
Con: subjective , interrator variability, interpersonal relationships can play a role
Checklists being developed to assess history taking, physical exam performance and information sharing (6,7)-effort to standardize encounters
Some researchers feel that since this is happening less and less it should be weighted less for grading (8)
Verbal feedback is difficult and requires training to be done effectively(9)
STRONG PUSH TO SYSTEMATIZE(10)

12. 6 core competencies
Developed by ACGME committees after Feds said they wanted better doctors graduated from residencies.
Now these competencies are applied to medical students

13. 6 core competencies-pro/con
Standardized form developed by committee to describe what a good doc should be able to do
Thought to be complete evaluation
Widely used
Not well understood outside of academic circles (and sometimes even in academic circles!)(11)

14. MINI-CEX
Direct observation of a specific part of a clinical interaction – ie-the knee exam

15. MINI-CEX Pro/con
Many studies show it is effective, students find it valuable(11)
Studies show good interrator reliability (11)
Requires TIME and energy and cooperative patients
Can cause severe performance anxiety in some students affecting performance

16. OSCE exams
Use of standardized patients is well documented for reliability(13,14)
Now used as part of USMLE exams

17. OSCE Pros-standardized, objective(15)
Thought that it may be useful for evaluation of 6 core competencies?(16)
Cons-expensive, time-intensive

18. RIME
Newer evaluation format:
Reporter
Interpreter
Manager
Educator

19. RIME Pro- it is new-supposed to be easier/ more reliable to use
Con-it is new-similar handicaps as 6 core competencies (wording, too easy to give everyone high marks)
“It is not focused on technical skills…the RIME scheme is used for students to show they were able to demonstrate skills, be consistent and progressive, and to show improvements” (17)

20. Clinical Logs
Used to track the number of patients seen and their diagnoses
Require good administrative skills on the part of the student
“If you talk to or touch the pt you should log the encounter” (18)
Many schools have a list of diagnoses the students must see in order to graduate for example-UNC96(19)

21. Presentation at Community Site

22. In Conclusion…
Clerkship students are being evaluated in many dimensions
Is it any better?
What are other programs doing?

23. How do they Evaluate Us?
“Involving students in a humanistic but rigorous approach to medicine and being a physician students wanted to emulate seem particularly important. These aspects appear potentially amenable to faculty development” (20)

24. THANKS!

25. References
1- Study: Clerkship grading varies “Dramatically” at U.S. Medical Schools, US News and World Report, july 12, 2012.
2-Variation and imprecision of clerkship grading in U.S. medical schools, Acad. Med., 2012 Aug,87(8):
3-Why Failing Med students Don’t Get Failing Grades, The New York Times, Feb 28, 2013.
4-medical school policies regarding struggling students during the internal medicine clerkships: results of a national survey- Acad. Med.-Sept 8 vol 83 issue 9 pp

26. Continued
5-”Making the Grade”- Noncognitive predictors of medical students’ clinical clerkship grades, J.Natl Med Assoc, 2007 oct; 99(10)
6- Direct observation by Faculty, Practical Guide to the Evaluation of Clinical Competence, Chap 9,
7- Direct Observation in Medical Education: A review of the Literature and Evidence for Validity, Mount Sinai School of Medicine, 29 July 2009, vol76, issue4 pp

27. More references
8- Direct Observation of Students during Clerkship Rotations: a Multiyear Descriptive Study. Acad Med, mar 2004, vol79 Issue 2, pp
9-Assessment in Medical Education, Clinical Collections NEJM, 2007,jan 25,2007, pp
10-Systematic Direct Observation of Clinical Skills in the Clinical Year, Mededportal.org feb13,2014, pub id 9712.
11-Competency and the six core competencies, JSLS, 2002, Apr-Jun; 6(2); 95-97

28. More references
12-Examiner Differences in the Mini-CEX; Advances in Medical Education, pp
13- An Overview of the uses of standardized patients for teaching and evaluating clinical skills; Acad Med, June 1993.
14-Use of an Objective Structured Clinical Examination in evaluating student performance; Family Medicine; May 1998 pp
15- OSCE- The assessment of choice; Oman Med J 2011 jul;26(4)

29. And more…
16-Is OSCE valid for evaluation of the six ACGME general Competencies? Science Direct; 74(2001)
17(RIME) for the Evaluation of First Professional Degree Students; Mededportal; id 834 Sept 12, 2013.
18-Clinical Log: UNC.EDU/medclerk.
19-Clinical Log-UNC.EDU/medclerk.
20-Third-year medical students’ perceptions of effective teaching behaviors in a multidisciplinary ambulatory clerkship; Acad Med, Aug2008; 78(8) p