1. Establishing Content Validity of a Novel Written Examination to Assess Medical Students on the Surgery Clerkship A Berlin MD, A Reinert BA, A Swan-Sein MEd PhD, R Nowygrod MD, A Fingeret MD

2. Agenda Needs assessment Exam Development Exam Design Methodology Results Discussion

3. Needs Assessment Historical assessment of students – Clinical evaluations – Unstructured oral exam – NBME surgery subject exam Goals of new evaluation tool – Objective and reliable – Congruent with educational objectives – Based on ACGME core competencies

4. Exam Design Novel Exam: Clinical Skills Exam (CSE) – 20+ clinical scenarios 5 randomly selected per exam – Short answer or essay response – Administered over 3 hours – ExamSoft computer software Grading – Rubric – Senior residents

5. Example 1 Identify operation Obtain surgical consent Determine order Describe steps shown

6. Example 2 Generate differential based on history Describe risk factors Interpret radiographic study Identify next step in management

7. Methodology July, 2011 - February, 2013 – n = 262 Analysis – Descriptive statistics – Multivariate analysis Comparison data for validity assessment – USMLE Step 1 and 2CK – NBME Subject Exams in Medicine and Surgery – Clerkship clinical evaluations – Clerkship order

8. Results: Descriptive Statistics – CSE Content and Score Distribution

9. Results: Trend Analysis – CSE and NBME Surgery Subject Exam

10. Results: Multivariate Statistics - Validity Analysis CSENBME Correlation Linear RegressionCorrelation Linear Regression rp valuer NBME Surgery0.36*0.82 NBME Medicine0.010.540.080.20 Step 10.34*0.820.55*0.02 Step 20.44*0.0070.56*0.004 Clerk - Clinical Reasoning 0.48*0.0010.43*0.09 Clerk - Fund of Knowledge 0.56*0.0040.63*< 0.001 * = p < 0.001

11. Results: Multivariate Statistics – Validity Analysis CSE NBME Bivariate Analysis Logistic Regression p value Clerkship orderANOVA0.30< 0.0010.01 OB/Gyn priorX20.34< 0.0010.02 Medicine priorX20.69< 0.0010.03

12. Results: Student Feedback on CSE

13. Results: Student Feedback - Word Cloud

14. Limitations – Study Generalizability Statistical analysis

15. Future Directions Future directions – New content – Open access – Computer algorithm based grading – Use as formative assessment mid-clerkship

16. Conclusions CSE is a useful adjunct to NBME surgery subject exam – Independent of clerkship order – Course goals and objectives – ACGME core competencies

17. Questions / Comments