1. Bridge over Troubled Water Linking UME and GME Monica L. Lypson, MD Assistant Dean, Graduate Medical Education University of Michigan Jeff Fabri, MD Rita M. Patel, MD

2. Education Across the Continuum Standardization of Assessment Student Portfolios to Baseline Assessment Monica L. Lypson, MD Assistant Dean, Graduate Medical Education University of Michigan Associate Chief of Staff, VA Ann Arbor Health Care System

3. Institutional OSCE – Post-Graduate Orientation Assessment (POA)  Developed by the Graduate Medical Education Committee (GMEC) at the University of Michigan  It is our initial step in training our residents  It was established to determine residents’ baseline proficiency in particular aspects of the ACGME’s six general competencies Lypson ML. et.al. Academic Medicine. 79(6):564-70, 2004 Jun.

4. POA CONTENT  Knowledge and skills needed during the first six to eighteen weeks of residency/internship  Emphasizes clinical situations that are often encountered without formal supervision  9 Assessment & Educational Stations

5. The POA as Formative Assessment  Results of the POA determine the basis for individualized “learning agendas”  Remediation is provided after the completion of each station  Residents received educational materials that provide the “answers” to the information assessed during the POA  Standardized Patient feedback is provided to the program director within 24 hours if the resident performs exceptionally well or poorly during the POA

6. Implementation  Administered over 4 days of Paid hospital orientation  Provided to approximately 150 PGY-1 residents in over 15 specialties  Cost Approximately $250 per resident for the assessment - $500 if you add in salary  There are some discipline specific scenarios – e.g. Pediatric cases and examples  Scores are provided to the resident and program director within 7-10 days of the POA

7. Subspecialty Programs Dentistry Emergency Medicine Family Medicine General Surgery –Preliminary Residents Internal Medicine –Preliminary Residents Internal Medicine – Pediatrics Neurosurgery Obstetrics & Gynecology Otolaryngology Orthopedics Pathology Pediatrics Plastic Surgery Psychiatry Urology PGY-2s (Residents that did not complete their Internships at UMHS) –Physical Medicine & Rehabilitation –Dermatology –Neurology

8. Critical Values / Multiple Choice –Computer Based assessment –Review and diagnose 17 short patient scenarios –Multiple Choice Format similar to United States medical Licensing Exam (USMLE) Step 3 –Competencies: Patient Care Medical Knowledge Socio-Cultural Communication –Standardized patient assessment –Assesses the understanding of disease and treatment recommendations in the context of the patient’s health beliefs and socio-economic setting –Competencies: Interpersonal and Communication Skills Professionalism Patient Care Practice-Based Learning and Improvement Station Content

9. Evidence-Based Medicine (EBM) –Computer-based Assessment –Generate a clinical question –Residents reviewed abstracts & identify the appropriate treatment –Competencies: Practice-based Learning and Improvement Medical Knowledge Images (X-Rays) –Computer-based Assessment –Review & diagnose 18 images –Many of the common films reviewed in the middle of the night while “on call” –Competencies: Patient Care Medical Knowledge

10. Station Content Informed Consent –Standardized patient Assessment –Obtain informed consent from a patient for a procedure –Hospital & JCAHO standards and policies –Competencies: Interpersonal and Communication Skills Professionalism Patient Care Patient Safety –Pen & Paper Assessment –Order Writing Station Legibility Signature, Date, Time –Respiratory Distress Treatment of the acute Asthmatic –Competencies: Systems-Based Practice Professionalism Patient Care Medical Knowledge

11. Station Content Aseptic Technique –Checklist Evaluation by Expert Nursing Staff and Standardized Patients –Create & maintain a sterile field while performing a “mock” I&D –Universal Protocol & “Time-out” procedures covered –Assessment/Remediation materials include a review of JCAHO requirements –Improved nursing and house officer interactions –Competencies: Patient Care Medical Knowledge Systems-Based Practice

12. Station Content System Compliance / Fire Safety Station –Surgical Fire Safety (20/20) –Housestaff involvement with a Patient fire –Questions covering use of safety equipment & review of JCAHO requirements –Video & Computer Based Assessment –Competencies: Patient Care Systems-Based Practice Pain Assessment –Pain Assessment Tools –Educates on the appropriate medications to use for pain –Explains hospital and JCAHO Pain assessment mandates –PowerPoint & Computer Based Assessment –Competencies: Patient Care Medical Knowledge

13. Resident Satisfaction with the POA -Survey Response Rate 93% 2002-2004 QuestionYES Have you learned any new clinical skills during this assessment? 70.6% Do you think this was a useful way to spend part of orientation? 83.4% Do you feel better prepared for some aspects of internship after this assessment? 80.6% Do you expect to refer back to the teaching materials handed to you today? 84.7% Would you recommend that we continue the POA next year? 84.7%

14. The POA was based on the following… The University of Michigan Comprehensive Clinical Assessment (CCA)

15. Comprehensive Clinical Assessment 1991-Present A high-stakes examination for University of Michigan Medical Students Fourth Year students must pass the CCA in order to Graduate 10-15 stations over ~4-5 hours ~150 M4 students per year Summative Evaluation Standards have been set for Pass/Fail Remediation for failures Rochester AB. et.al.. Academic Radiology. 5(3):169-72, 1998

16. Summary of Station Comparison: CCA & POA POA- GME Formative Informed Consent & Policy Pain Assessment Socio-Cultural Communication Evidence Based Medicine Images CCA-UGME Summative History & Physical Diagnosis Geriatrics Socio-Cultural Communication Evidence Based Medicine Images

17. Institutional Implications for GME & UGME Training the Faculty in the Teaching Skills of competency assessment –Medical School Objectives Project (MSOP) –Accreditation Council of Graduate Medical Education Reimbursement of faculty time and effort Payment for both assessments – Medical School vs. Hospital –Is the medical school or the hospital responsible for ‘educational cost’? Trainee awareness of “competency gap” between the expectations of medical School and residency

19. POA Demographics 2002-2004 Gender Male54% Race White / Non-Hispanic Underrepresented Minorities Other 70.1% 6.5% 23.4% Medical Schools Public International 62% 1.5% United States Medical Licensing Examination (USMLE) Step 1 Step 2 231 233

20. The University of Michigan Health System Plan for Competency Based Resident Education Joint Hire faculty member with the Department of Medical Education Centralized OSCE – Post-Graduate Orientation Assessment (POA) This is based on the system already in place for Undergraduate Medical Education at UMHS.

21. The Department of Medical Education Chairman: Larry Gruppen, PhD One of ~5 departments of medical education at LCME accredited medical schools The department had focused on CME and UGME in the past with assessment and research expertise The office of Graduate Medical Education and the Department of Medical Education –Joint Hire –Job Description 50% appointment GME office, 50% appointment Dept. of Med. Ed. Aid Residency programs with curriculum development, core competency assessment Use this work as a vehicle of scholarly pursuit –The department has a long history of UGME – now will develop its expertise in GME