ACGME & OSCE Use of OSCE in Residency Evaluations Paul Gordon, MD EOSG May 2003

ACGME Toolbox OSCE is listed as the most desirable for patient care interpersonal and communication skills professionalism next best method for practice-based learning and improvement systems-based practice The toolbox suggests written and oral exams as the most desirable for assessing medical knowledge.

But… OSCE measures have not traditionally been designed to take into account levels of expertise creating better OSCEs require understanding of how interviewing style of experts differs objectively from novices

Continued Problems OSCEs have traditionally employed binary content checklists to score behaviors demonstrated by students growing recognition of the inability of checklists to capture complex human behaviors such as empathy and organization has resulted in the adoption of multipoint global ratings designed to capture non-binary characteristics of the performance

Students vs. Residents professionals pass through five stages: novice, advanced beginner, competence, proficiency and expertise early novice stage is characterized by the collection of large amounts of data, in no particular order, with little regard for situational factors, which are then used to synthesize a problem solution

Students vs. Residents Experts: gather much more focused information rely on many different types of data including situational variables compare all observations to previous experiences and; quickly and automatically respond to such observations, often without resorting to any formal process of problem solving

Students vs. Residents experts cannot easily break down their thinking into component steps and therefore have great difficulty returning to a novice form of solving problems checklists penalize clinicians who arrive at diagnoses quickly as a result of pattern recognition and rapid hypothesis testing

What to Do? need to look at variables such as types and sequence of questions and statements measures that are sensitive to the nature of expertise should include the sequence and organization rather than just the number of questions asked

What to Do? while there appear to be patterns of interviewing that characterize different levels of expertise, they are subtle and may not be amenable to counting and classification for examination purposes

Interviewer Utterances questions summary statements empathic comments articulated transitions information giving

Research Mean number of utterances exceeded one every 10 seconds for all groups largest proportion was Questions 76% for clerks; 67% for experts 1/3 of total utterances consisted of a group of “low frequency” types: empathic comments, information giving and summary statements

Research While utterance type over time appeared to show characteristic patterns reflective of expertise, the differences were not robust only the pattern of use of Summary Statements was statistically different between groups (P<0.05)

Summary Residents are different than students Measures that are sensitive to the nature of expertise should be used to supplement OSCE checklists that simply count questions Information giving, empathic comments and summary statements should be credited While there appear to be patterns of utterances that characterize levels of expertise, the patterns are subtle and not always amenable to counting