Performance on Brief Practice Exam Identifies residents at Risk for Poor ABSITE and ABS Qualifying Exam Performance Michael Corneille MD, Ross Willis PhD, Ronald Stewart MD, Daniel Dent MD

Background ABSITE First offered 1975 to gauge residents’ acquisition of fundamental surgical knowledge Not designed as predictive tool Strong correlation between performances on ABSITE and ABS QE Shellito et al. (2010) Am J Surg No pass/fail cutoff ABSITE vs ABS QE pass rates ABSITE >30 th %ile - 93% 1st attempt QE passed ABSITE ≤30 th %ile - 73% 1st attempt QE passed 2009 ABSITE.51 correlation between score and QE score (1 st time).92 correlation between score and QE pass (1 st time)

Act as a study tool Identify residents at risk for poor performance on ABSITE and QE 2 part exercise Constructing the exam Taking the exam Objectives based on potential reasons for poor performance Purpose of Practice Exams

ObjectivesHow goal met by practice exam Understanding that the exam comprised a finite list of topic areas. Distribution of ABSITE keywords Master the 3-5 topics resident was assigned.Writing questions Develop insight into how a test is constructed.Writing questions Encourage resident to focus on the “board relevant” material. Writing questions Understand that it is possible to prepare for the exam. Test taking Identify residents earlier than March who will need help. Test taking Serve as a wake-up call to resident that his/her study method is not effective and his/her performance needs to improve. Test taking Keep residents engaged in continuous studying for the ABSITE. Offering exam various times throughout the year

Background Practice test not designed as a predictive tool Same guidelines as ABSITE Scores not used to determine promotion Allow program director to objectively compare resident’s fund of knowledge to peers

Hypothesis We lacked objective data to counsel residents as to true interpretation of practice test scores Hypothesis A brief practice exam May identify residents at risk of poor ABSITE performance May identify early in the chief resident year those at risk for poor QE performance.

Methods Study period RRC approved to graduate 9 chiefs/year 1-3 practice exams/year 60 mins/exam (50-55 basic science & clinical questions) Questions based on the ABSITE keywords authored by residents and edited by faculty Assigned 3 to 5 topics Standardized format Question 5 response options Explanations with references Optically graded

Methods Exam was considered mandatory Educational time allotted within the training program Scores reported to residents within two weeks % correct and standard deviation within PGY class Residents >1 standard deviation below the mean for their class counseled Expectation of the program > 30th %ile reiterated Referred to a faculty mentor to develop study plan

Data Analysis Compared resident’s practice exam most recently preceding ABSITE to that year’s ABSITE Compared chief year ABSITE to ABS QE Pass/fail status Spearman rank correlation coefficients used to compare dependent measures

Results 9 practice exams administered to all resident levels 432 individual practice exams completed 309 residents have taken the ABSITE 33 have taken the ABS QE 263 residents completed at least one practice exam in the year preceding the ABSITE 26 chief residents completed at least one practice exam immediately preceding the ABS QE

ABSITE YearNr (Spearman)Significance p < p < p < p < p <.01 Practice Test vs ABSITE

Correlation between Practice Exam and first attempt QE score was significant (r =.416, p <.05) Correlation between chief year ABSITE and the first attempt QE score was significant (r =.743, p <.01) ABSITE score <30 th %ile 89% passed the QE on the first attempt Data Analysis

Discussion What to do with the data? Can we intervene? Is the exercise beneficial?

Several authors have reported that intervention can improve scores 1,2 Our data show that performance on practice exam is correlated to ABSITE and ABS QE May suggest that our interventions are inadequate or that there is a cohort of residents destined to struggle with standardized assessments Discussion 1 Borman KR, Does Academic Intervention Impact ABS Qualifying Examination Results? Current Surgery, Vol 63, Issue 6, Pp , Nov Hirvela ER, Becker DR Impact of Programmed Reading on ABSITE Performance. The American Journal Of Surgery, Vol 162 Nov 1991)

Discussion Silver lining May seem overall pessimistic Those who do struggle may be excellent surgeons Those who struggle may pass their boards on the first attempt 89% first attempt pass with ABSITE scores less than 30th percentile

Conclusions Poor performance on a brief practice exam Does identify residents at risk for poor ABSITE performance Does identify residents at risk for poor ABS QE performance and earlier than ABSITE scores.

Conclusions More data is necessary on what interventions truly work to improve ABS QE pass rates in very high risk residents More data is necessary on what interventions truly work to improve ABS QE pass rates in very high risk residents

Thank You