End of Shift Milestone Evaluation Forms Michael S. Beeson, M.D., MBA Program Director, Akron General Medical Center Amber Bradford, D.O. Associate Program Director, Akron General Medical Center Steven Warrington, M.D. HAENOW Milestone Study Group Mike

HAENOW? Hennepin- Danielle Hart, M.D. Akron General- Beeson, Bradford, Warrington Einstein- (Beth Israel Albert Einstein College of Medicine) Saadia Akhtar, M.D. Nevada- Michael Epter, D.O. Ohio State- Diane Gorgas, M.D. Washington- Fiona Gallahue, M.D. Mike

Objectives of Session Introduce concept of End of Shift Milestone Evaluation Forms Understand advantages/drawbacks of this method Introduce elements of reliability and validity of this method Mike

Components of Session Milestone assessment basics Development of End of Shift Milestone Evaluation Forms Description of End of Shift Milestone Evaluation Forms Viewing of resident-patient interaction Scoring of 1 of 8 EM1 End of Shift Evaluation Forms Discussion of this Methodology Concluding remarks Mike

We Got Milestones, Now What? Assessment Objective rather than subjective Mike

Core Competencies Introduction Formative vs Summative assessment Mike

Milestone Assessment Focus on Reliability and Validity of methods and tools Will different faculty rate the same? Is the setting valid? Is the assessment instrument valid? How is bias controlled? Mike

Emergency Medicine is Different IM/Surgery rotations with same senior residents and rounding attending(s) EM shifts can be with a different attending EVERY shift For EM, difficult to do an end of month Milestone evaluation An attending may have worked with a specific resident sparingly Any assessment becomes a subjective global rating scale Mike- We abused global ratings scales with measurement of the core competencies, often using an end of month or end of six month evaluation of a resident. RMS facilitated this, and used of Likert scales for resident assessment.

General Milestone Assessment Basics The farther from patient care (real or simulated) an assessment is made, the more it resembles a subjective global ratings scale The closer to direct patient care (real or simulated) assessment is made the more objective Mike

Turning the EM Milestones into Global Rating Scales Bias exists from faculty matching PGY level with expected proficiency level Mike

End of Shift Milestone Evaluation Forms Struggled with how to make an efficient end of shift evaluation form SDOT useful, but could take 20-30 minutes to complete Desire to develop a brief end of shift assessment tool Novel concept- make it useful too! Mike

Milestones to Exclude Procedure-based (PC9- PC14) Medical Knowledge (MK) Milestones left: Milestone Number by Proficiency Level Level 1 2 3 4 Total 23 26 40 31 120 Mike

Separate Forms for EM1 and EM2-4 EM1- Use Milestones from Proficiency levels 1-3 89 Milestones EM2-4 Use Milestones from Proficiency levels 2-4 97 Milestones 8 EM1 forms and 8 EM2-4 forms Proficiency levels of each Milestone are NOT identified Amber

Sample End of Shift Milestone Form Amber

Sample End of Shift Milestone Form with Anchors Amber

Process Milestone evaluation training with individual faculty A separate form is used on a weekly basis Program Coordinator swaps them out weekly Weekly email of that week’s End of Shift Evaluation Forms along with their anchors to faculty and residents Towards end of shift a form is completed by the attending with the resident Form is then faxed to Residency Office by ED Unit Clerk Amber

Let’s Score an Intern View video Amber

Additional Case Points Intern does not introduce himself to the patient Intern addresses patient by the wrong name Intern was 10 minutes late to his shift and was told to wear his white coat to work the day before Intern does not perform a head or neck exam but communicates a normal exam Intern does not communicate elevated blood pressure to his attending Amber

Case Points Intern initially prescribes Percocet to a patient with a known alcohol problem Intern initially wants to place the patient in an air cast splint for a distal fibular fracture Intern initially refers the patient to the medicine clinic for follow-up Amber

Score the End of Shift Evaluation Form Amber

Data Input of Each Form Each form is faxed to the residency office Program Coordinator inputs form into online database Demonstration of Form data input Mike

Reports from Data Demonstration of Resident Report Mike

Issues to be Worked Out Better balancing of each form in terms of Proficiency levels Which Milestones are just too difficult to evaluate using this method Could procedural subcompetencies be evaluated this way? Airway, Wound Care, US, Central Line Forms? Form completion compliance Buy-in by residents and faculty Unknown Inter-Rater Reliability Mike

State of This System Available now! www.cordtests.org Forms can be downloaded as Word document, put your logo on it, whatever… iPad app is nearly completed Mike