1. End of Shift Milestone Evaluation Forms
Michael S. Beeson, M.D., MBA
Program Director, Akron General Medical Center
Amber Bradford, D.O.
Associate Program Director, Akron General Medical Center
Steven Warrington, M.D.
HAENOW Milestone Study Group
Mike

2. HAENOW? Hennepin- Danielle Hart, M.D.
Akron General- Beeson, Bradford, Warrington
Einstein- (Beth Israel Albert Einstein College of Medicine) Saadia Akhtar, M.D.
Nevada- Michael Epter, D.O.
Ohio State- Diane Gorgas, M.D.
Washington- Fiona Gallahue, M.D.
Mike

3. Objectives of Session
Introduce concept of End of Shift Milestone Evaluation Forms
Understand advantages/drawbacks of this method
Introduce elements of reliability and validity of this method
Mike

4. Components of Session Milestone assessment basics
Development of End of Shift Milestone Evaluation Forms
Description of End of Shift Milestone Evaluation Forms
Viewing of resident-patient interaction
Scoring of 1 of 8 EM1 End of Shift Evaluation Forms
Discussion of this Methodology
Concluding remarks
Mike

5. We Got Milestones, Now What?
Assessment
Objective rather than subjective
Mike

6. Core Competencies Introduction
Formative vs Summative assessment
Mike

7. Milestone Assessment
Focus on Reliability and Validity of methods and tools
Will different faculty rate the same?
Is the setting valid?
Is the assessment instrument valid?
How is bias controlled?
Mike

8. Emergency Medicine is Different
IM/Surgery rotations with same senior residents and rounding attending(s)
EM shifts can be with a different attending EVERY shift
For EM, difficult to do an end of month Milestone evaluation
An attending may have worked with a specific resident sparingly
Any assessment becomes a subjective global rating scale
Mike- We abused global ratings scales with measurement of the core competencies, often using an end of month or end of six month evaluation of a resident. RMS facilitated this, and used of Likert scales for resident assessment.

9. General Milestone Assessment Basics
The farther from patient care (real or simulated) an assessment is made, the more it resembles a subjective global ratings scale
The closer to direct patient care (real or simulated) assessment is made the more objective
Mike

10. Turning the EM Milestones into Global Rating Scales
Bias exists from faculty matching PGY level with expected proficiency level
Mike

11. End of Shift Milestone Evaluation Forms
Struggled with how to make an efficient end of shift evaluation form
SDOT useful, but could take minutes to complete
Desire to develop a brief end of shift assessment tool
Novel concept- make it useful too!
Mike

12. Milestones to Exclude Procedure-based (PC9- PC14)
Medical Knowledge (MK)
Milestones left:
Milestone Number by Proficiency Level
Level 1 2 3 4
Total 23 26 40 31
120
Mike

13. Separate Forms for EM1 and EM2-4
EM1- Use Milestones from Proficiency levels 1-3
89 Milestones
EM2-4 Use Milestones from Proficiency levels 2-4
97 Milestones
8 EM1 forms and 8 EM2-4 forms
Proficiency levels of each Milestone are NOT identified
Amber

14. Sample End of Shift Milestone Form
Amber

15. Sample End of Shift Milestone Form with Anchors
Amber

16. Process Milestone evaluation training with individual faculty
A separate form is used on a weekly basis
Program Coordinator swaps them out weekly
Weekly of that week’s End of Shift Evaluation Forms along with their anchors to faculty and residents
Towards end of shift a form is completed by the attending with the resident
Form is then faxed to Residency Office by ED Unit Clerk
Amber

17. Let’s Score an Intern
View video
Amber

18. Additional Case Points
Intern does not introduce himself to the patient
Intern addresses patient by the wrong name
Intern was 10 minutes late to his shift and was told to wear his white coat to work the day before
Intern does not perform a head or neck exam but communicates a normal exam
Intern does not communicate elevated blood pressure to his attending
Amber

19. Case Points
Intern initially prescribes Percocet to a patient with a known alcohol problem
Intern initially wants to place the patient in an air cast splint for a distal fibular fracture
Intern initially refers the patient to the medicine clinic for follow-up
Amber

20. Score the End of Shift Evaluation Form
Amber

21. Data Input of Each Form Each form is faxed to the residency office
Program Coordinator inputs form into online database
Demonstration of Form data input
Mike

22. Reports from Data
Demonstration of Resident Report
Mike

23. Issues to be Worked Out
Better balancing of each form in terms of Proficiency levels
Which Milestones are just too difficult to evaluate using this method
Could procedural subcompetencies be evaluated this way?
Airway, Wound Care, US, Central Line Forms?
Form completion compliance
Buy-in by residents and faculty
Unknown Inter-Rater Reliability
Mike

24. State of This System Available now!
Forms can be downloaded as Word document, put your logo on it, whatever…
iPad app is nearly completed
Mike