1. ACGME & OSCE Use of OSCE in Residency Evaluations
Paul Gordon, MD
EOSG
May 2003

2. ACGME Toolbox OSCE is listed as the most desirable for
patient care
interpersonal and communication skills
professionalism
next best method for
practice-based learning and improvement
systems-based practice
The toolbox suggests written and oral exams as the most desirable for assessing medical knowledge.

3. But…
OSCE measures have not traditionally been designed to take into account levels of expertise
creating better OSCEs require understanding of how interviewing style of experts differs objectively from novices

4. Continued Problems
OSCEs have traditionally employed binary content checklists to score behaviors demonstrated by students
growing recognition of the inability of checklists to capture complex human behaviors such as empathy and organization has resulted in the adoption of multipoint global ratings designed to capture non-binary characteristics of the performance

5. Students vs. Residents professionals pass through five stages:
novice, advanced beginner, competence, proficiency and expertise
early novice stage is characterized by the collection of large amounts of data, in no particular order, with little regard for situational factors, which are then used to synthesize a problem solution

6. Students vs. Residents Experts: gather much more focused information
rely on many different types of data including situational variables
compare all observations to previous experiences and;
quickly and automatically respond to such observations, often without resorting to any formal process of problem solving

7. Students vs. Residents
experts cannot easily break down their thinking into component steps and therefore have great difficulty returning to a novice form of solving problems
checklists penalize clinicians who arrive at diagnoses quickly as a result of pattern recognition and rapid hypothesis testing

8. What to Do?
need to look at variables such as types and sequence of questions and statements
measures that are sensitive to the nature of expertise should include the sequence and organization rather than just the number of questions asked

9. What to Do?
while there appear to be patterns of interviewing that characterize different levels of expertise, they are subtle and may not be amenable to counting and classification for examination purposes

10. Interviewer Utterances
questions
summary statements
empathic comments
articulated transitions
information giving

11. Research
Mean number of utterances exceeded one every 10 seconds for all groups
largest proportion was Questions
76% for clerks; 67% for experts
1/3 of total utterances consisted of a group of “low frequency” types:
empathic comments, information giving and summary statements

12. Research
While utterance type over time appeared to show characteristic patterns reflective of expertise, the differences were not robust
only the pattern of use of Summary Statements was statistically different between groups (P<0.05)

13. Summary Residents are different than students
Measures that are sensitive to the nature of expertise should be used to supplement OSCE checklists that simply count questions
Information giving, empathic comments and summary statements should be credited
While there appear to be patterns of utterances that characterize levels of expertise, the patterns are subtle and not always amenable to counting