Using an OSCE Station to Evaluate Medical Student Performance in Telephone Communication with Patients

General Protocol for OSCE Station Development and Evaluation of Student Performance Key Course Concepts &  Case Creation  Observation of Evaluating the &  Case Creation  Observation of Evaluating the General Student-Patient  Encounter Competencies Encounters a) Content (H & P Items) --History Taking skills b) Technique (Communication --History Taking skills b) Technique (Communication --Communication Skills and Physical Exam Skills) --Communication Skills and Physical Exam Skills) --Physical Exam Skills --Physical Exam Skills ↓ Student Creates Evaluating the Student Creates Evaluating the Post-Encounter  Student Note Post-Encounter  Student Note Note a) Content (History Items, Note a) Content (History Items, Physical Exam Items, Physical Exam Items, Assessment, & Plan) Assessment, & Plan) b) Technique (Organization & b) Technique (Organization & Legibility) Legibility)

General Responsibilities in OSCE Station Construction 1.Determine key Course Concepts and General Competencies to be assessed Faculty Content Expert Faculty Content Expert 2.Case Creation Faculty Content Expert Faculty Content Expert Standardized Patient (S.P.) Program Team Standardized Patient (S.P.) Program Team

General Responsibilities in OSCE Station Construction—Continued 3.Standardized Patient Selection/Training and Station Logistics Standardized Patient (S.P.) Program Team Standardized Patient (S.P.) Program Team 4.Observer Rating Form Construction Faculty Content Expert Faculty Content Expert Standardized Patient (S.P.) Program Team Standardized Patient (S.P.) Program Team

General Responsibilities in OSCE Station Construction—Continued 5.Model S.O.A.P. Note Creation Faculty Content Expert Faculty Content Expert 6.Model S.O.A.P. Note with Key Words Identified Faculty Content Expert Faculty Content Expert 7.S.O.A.P. Note Rating Form Faculty Content Expert Faculty Content Expert SOAP Note Rater Training Staff SOAP Note Rater Training Staff

General Responsibilities in OSCE Station Construction—Continued 8.Validation/Evaluation of Processes Faculty Content Expert Faculty Content Expert Standardized Patient (S.P.) Program Team Standardized Patient (S.P.) Program Team SOAP Note Rater Training Staff SOAP Note Rater Training Staff

The OSCE Telephone Case “Sore Throat and Rash” A child with a history suggestive of Scarlatina

To be Evaluated—Student Mastery of Key Course Concepts and Achievement of General Competencies Key Course Concepts A)Recognition and management—acute strep throat and its complications. Taught In: 1. Basic Science Courses 2. Year 1 Clinical Medicine Course 2. Year 1 Clinical Medicine Course 3. Year 3 Family Medicine and Pediatric Clerkships 3. Year 3 Family Medicine and Pediatric Clerkships B)Managing telephone encounters with patients/families. Taught In: 1. Year 3 Pediatric Clerkship General Competencies A)History Taking Skills Taught In: 1. Year 1 Clinical Medicine Course 2. Year 2 Clinical Medicine/Physical Diagnosis Course 3. Year 3 Clerkships (All Disciplines) B)Communication Skills Taught In: 1. Year 1 Clinical Medicine Course 2. Year 2 Clinical Medicine/Physical Diagnosis Course 3. Year 3 Clerkships (All Disciplines)

Telephone Case Creation 1.Establish the Scenario/“Chief Complaint” Setting for encounter, e.g. Caregiver calls clinic answering service for advice. Setting for encounter, e.g. Caregiver calls clinic answering service for advice. Rationale for encounter, e.g. Why is she calling today (after normal clinic hours) Rationale for encounter, e.g. Why is she calling today (after normal clinic hours)

Telephone Case Creation 2.Identify critical content to be incorporated into SP Training Materials, and “Observer” Rating Form History of Present Illness, pertinent to the Chief Complaint of sore throat and rash History of Present Illness, pertinent to the Chief Complaint of sore throat and rash Contributory items from past medical, family, and social histories Contributory items from past medical, family, and social histories Pertinent negatives Pertinent negatives

Telephone Case Creation 3.Communication Skills to be assessed: Are identified and incorporated into Rating Form. Skills include: Are identified and incorporated into Rating Form. Skills include: Appropriate interaction with patient’s caregiver via telephone, e.g. Appropriate interaction with patient’s caregiver via telephone, e.g. Clarity of verbal communication, Clarity of verbal communication, Verification of information exchanged, Verification of information exchanged, Interviewing techniques, e.g. logical flow of questioning Interviewing techniques, e.g. logical flow of questioning Courtesy Courtesy

Telephone Case Creation 4.Specific desirable and undesirable responses to caller’s scripted questions identified and incorporated into Rating Form, e.g. SPs coached to ask student to call pharmacy with prescription SPs coached to ask student to call pharmacy with prescription Desirable student response: Child needs to be seen in clinic or emergency department Desirable student response: Child needs to be seen in clinic or emergency department Undesirable student response: Student agrees to call pharmacy Undesirable student response: Student agrees to call pharmacy

Telephone Case Creation 5.Create rating form scoring guide Criteria for each checklist item is given, including, as appropriate: Criteria for each checklist item is given, including, as appropriate: Detailed description of checklist item Detailed description of checklist item Examples of appropriate student responses to obtain credit for checklist item Examples of appropriate student responses to obtain credit for checklist item Examples of inappropriate responses Examples of inappropriate responses

Standardized Patient Selection/Training SP Selection, Preferably: Female – caregiver can be mother or grandmother for this caseFemale – caregiver can be mother or grandmother for this case Experience with children of same age as patientExperience with children of same age as patient Can function in role, e.g. can hear and understand student on telephoneCan function in role, e.g. can hear and understand student on telephone Can demonstrate ability to follow directionsCan demonstrate ability to follow directions

Standardized Patient Selection/Training SP Training procedures Materials distributed to SPs prior to first meeting. SPs are expected to review materials in advance.Materials distributed to SPs prior to first meeting. SPs are expected to review materials in advance. All SPs trained to both portray patient role, and complete scoring checklist.All SPs trained to both portray patient role, and complete scoring checklist. During student runs, SPs work in pairs, one portrays parent, one completes checklistDuring student runs, SPs work in pairs, one portrays parent, one completes checklist SPs swap roles periodicallySPs swap roles periodically Reduces SP fatigue, improves reliabilityReduces SP fatigue, improves reliability

Standardized Patient Selection/Training SP Training procedures, continued Example first meeting with SPs:Example first meeting with SPs: Review patient and scoring materials.Review patient and scoring materials. Practice patient portrayals, one SP plays parent while others score student performance.Practice patient portrayals, one SP plays parent while others score student performance. Score sheets reviewed as a group, discrepancies explored and clarified as neededScore sheets reviewed as a group, discrepancies explored and clarified as needed Second, then third SPs portray patient while others score student performancesSecond, then third SPs portray patient while others score student performances Score sheets reviewed and fine tuned after each practiceScore sheets reviewed and fine tuned after each practice

Standardized Patient Selection/Training SP Training procedures, continued Example second meeting with SPs:Example second meeting with SPs: Recap of previous session.Recap of previous session. Practice patient portrayals as before, allowing remaining SPs opportunity to portray parent while others score student performance.Practice patient portrayals as before, allowing remaining SPs opportunity to portray parent while others score student performance. Score sheets reviewed and further fine tuned after each practiceScore sheets reviewed and further fine tuned after each practice Dry runs may be integrated into second sessionDry runs may be integrated into second session

Station Logistics Physical Needs: Rooms with functioning telephone connectionRooms with functioning telephone connection Ability to synchronize with central timerAbility to synchronize with central timer If in same exam area, close enough to easily hear announcements, but not close enough for students to overhear, e.g. next roomIf in same exam area, close enough to easily hear announcements, but not close enough for students to overhear, e.g. next room If in different location, perhaps a separate link to centralized controlIf in different location, perhaps a separate link to centralized control Sufficient, and spare supplies in SP roomSufficient, and spare supplies in SP room Enough trained personnel to ensure frequent relief breaks for SPsEnough trained personnel to ensure frequent relief breaks for SPs

Station Logistics Physical Needs: Clear, unambiguous instructions for students and SPs, including contingency plan instructions in case of “technical difficulties.”Clear, unambiguous instructions for students and SPs, including contingency plan instructions in case of “technical difficulties.”

Station Logistics Ensuring good exam day flow: SPs need to practice using telephone set, in the exam settingSPs need to practice using telephone set, in the exam setting SPs need to verify they are easily able to hear timer / hall monitor instructionsSPs need to verify they are easily able to hear timer / hall monitor instructions SPs need to be familiar with contingency plans in the event of problemsSPs need to be familiar with contingency plans in the event of problems

Results Mean Min2 5 Max27 28 Max Avail33 Std Dev Flags* *Flag = Number of students who complied with mother’s request to call pharmacy with prescription.

Implications/Discussion Standard, “bread-and-butter” Family Medicine scenario that students are expected to manage appropriately Standard, “bread-and-butter” Family Medicine scenario that students are expected to manage appropriately Easy case to prepare and present for students Easy case to prepare and present for students Case construction format allows new case scenarios to be quickly developed using this case as a model Case construction format allows new case scenarios to be quickly developed using this case as a model Because station is SP rated, Because station is SP rated, Training sessions need to include enough practice with checklist completion. Training sessions need to include enough practice with checklist completion. Clear checklist completion guidelines are imperative Clear checklist completion guidelines are imperative

Implications/Discussion Anecdotal evidence suggests knowledge of station by at least some students taking exam in Anecdotal evidence suggests knowledge of station by at least some students taking exam in Fewer students “caught in trap” of agreeing to call pharmacy with prescription in 2005 (12 vs 112 in 2004) Fewer students “caught in trap” of agreeing to call pharmacy with prescription in 2005 (12 vs 112 in 2004) Overall station scores were similar between presentations of case in 2004/05 Overall station scores were similar between presentations of case in 2004/05 Scores on communications skills items generally high Scores on communications skills items generally high Generally “hit and miss” on 9 Associated Symptoms items Generally “hit and miss” on 9 Associated Symptoms items

Please feel free to contact us  Maurice Kavanagh  Tom Roe