1. American Board of Surgery Examinations: Can We Identify Surgery Residency Applicants Who Will Pass the Examinations on the First Attempt? John L. Shellito, M.D., Jacqueline S. Osland, M.D., Stephen D. Helmer, Ph.D., Frederic C. Chang, M.D. Department of Surgery University of Kansas School of Medicine – Wichita

2. Introduction: The Importance of Board Certification One of the key demonstrations of the effectiveness of a residency’s training program ACGME Program Requirements-Surgery –65% first attempt pass rate for both the qualifying and certifying exam

3. Objective This study was performed to identify and evaluate factors which predict successful completion of both the American Board of Surgery Qualifying Exam (QE) and Certifying Exam (CE) on the first attempt

4. Methods Retrospective review –University of Kansas SOM-Wichita Surgery Residency Program –Included residents entering the program from 1990-2001

5. Pre-Admission Measures Age and gender Class rank AOA status Honors in third year surgery clerkship Interview scores National Board of Medical Examiners (NBME) Part 1 and Part 2 scores or United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores Rank list number

6. Post-admission Measures American Board of Surgery In Training Examination (ABSITE) scores Resident awards (Research, Teaching, Outstanding Senior Resident) Faculty evaluations of the 5 th year residents

7. Statistical Analysis Univariate analyses were performed using Fischer’s exact test for the dichotomous variables (i.e. gender, AOA status, honors) Two-sample Wilcoxon rank-sum test was used for continuous variables (i.e. scores, age) and ordinal data Logistic regression analysis was performed on the pre-admission variables

8. Results 77 medical students and residents –15 subjects excluded 5 NBME or USMLE scores not available 8 Left the program 2 never attempted the QE (both plastic surgeons) 62 subjects remained after exclusions

9. UKSM-W Surgery Residency Program 1990-2001 >90% of our graduates attended medical school in the Midwest (50% from KU) –One foreign medical graduate 16% (10/62) AOA Pass rates –First time: 77.4% –Eventual: 96.8% 2/62 (plastic surgery) Majority entered private practice (73%) No difference in first time pass rates over the study period

10. Results No significant difference based on gender 50 (80.7%) were male First time pass rate –74% for males/ 91.7% for females (p=0.179) Residents who successfully passed were slightly younger at time of application 27.1 years vs. 28.6 years (p=0.047)

11. Effect of Class Rank on First Attempt Pass Rates Class Rank GroupNumberPercent Top 1/3 rd 22/2395.7% Middle 1/3 rd 14/2263.6% Bottom 1/3 rd 3/560.0% P=0.0073

12. Effect of Step 1 scores on First Attempt Pass Rates Step 1 Group NumberPercent Top 50% (>200) 36/4285.7% Bottom 50% (<200) 12/2060.0% P=0.0247

13. Effect of Step 2 scores on First Attempt Pass Rates Step 1 GroupNumberPercent Top 3 Quartiles (>186) 44/5580.0% Bottom Quartile (<186) 2/540.0% P=0.0447

14. Results Cumulative scores for step 1 and 2 –>382 (top 2/3rds) 81.1% first time pass rate –<382 (bottom 1/3rd) 42.9% first time pass rate

15. Results Logistic regression analysis showed Step 1 scores to be the only independent predictor from the group of pre-admission factors for first time pass rates

16. Effect of ABSITE on First Attempt Pass Rates ABSITE Group NumberPercent PGY-1 >50 th percentile 41/5082.0% <50 th percentile 5/10 50.0% P=0.044 PGY-3 >50 th percentile 36/4090.0% <50 th percentile 12/22 54.5% P=0.002

17. Effect of ABSITE on First Attempt Pass Rates ABSITE GroupNumberPercent PGY-4 Top 1/3 rd 19/2286.4% Middle 1/3 rd 23/2688.5% Bottom 1/3 rd 6/14 42.9% P=0.009 PGY-5 Top 1/3rd23/2495.8% Middle 1/3rd15/2171.4% Bottom 1/3rd10/17 58.8% P=0.004

18. Results Residents winning an award during residency had a higher first time pass rate –92% versus 67% Faculty evaluations of 5 th year residents did not predict board success

19. Results Pre-admission factors that failed to predict success for first time pass rates –Honors in 3 rd year surgery clerkship –AOA status 10/10 first time pass rate, p=0.061 –Interview score –Ranking in the match

20. Discussion Clerkship grades Honors grade in third year surgery clerkship Dean’s letters Letters of recommendations Interview Score Class rank USMLE scores

21. Discussion Xu 1998 –Positive correlation between eventual board certification status and past academic performance –Included other surgical specialties in addition to general surgery Silber 2005 –Step 2 score single most important predictor for eventual OB/GYN board certification –Graduates from a single medical school, multiple residency programs

23. Summary This is first study to examine factors predicting first time board certification rates for general surgery residency graduates for both the QE and CE

24. Limitations Small sample size Bias injected from our selection process –No information about the success or failure of applicants who were excluded by our selection committee Single residency training program

25. Summary Academic aptitude by itself does not predict the multifaceted state of competence of a surgeon Professionalism, manual dexterity, communication abilities, and integrity are not measured by standardized tests Sorting applicants that are above a certain threshold of intellectual capacity solely by test scores may not be a good way to obtain the best residents

26. Conclusion Our study indicates a positive correlation between class rank, USMLE scores, ABSITE scores and first time pass rates for ABS certification