Is The ACGME Resident/Fellow Survey a Valid Tool to Assess General Surgery Residency Programs Compliance with Work Hours Regulations? Robert P. Sticca, MD, FACS Jay M. MacGregor, MD Randolph E. Szlabick, MD, FACS Department of Surgery University of North Dakota SMHS

Background 2003 – ACGME work hours regulations implemented 2004 – ACGME Residents/Fellows Survey 2008 – Special Message from CEO ACGME (Thomas Nasca, MD) –Standard Approach to Programs Across All Specialties with Potential Duty Hours Violations Identified in the Resident Survey

ACGME Special Message The Monitoring Committee has accepted the Resident Survey as a reliable, sensitive and reasonably specific tool The Monitoring Committee has set reasonable thresholds for categorization of a program as having Potential Duty Hours Violations (PDHV) based on the Resident Survey The validity of the Resident Survey has been demonstrated though significant correlations with site visit findings to warrant added scrutiny for programs meeting the thresholds of potential noncompliance found in the group targeted for follow-up. There is a significant correlation between resident reported violations of duty hours with deficiencies in other important areas of the learning environment (correlation 0.51, p<0.0001) Nasca, T.

ACGME Algorithm for Noncompliant Programs Nasca, T.

UND Experience 2005 – RRC site visit – additional categorical resident requested –5 year approval, no deficiencies –Additional resident denied, insufficient educational justification 2007 – RRC site visit – additional categorical resident requested –5 year approval, no deficiencies –Additional resident approved 2009 – RRC site visit – mandatory due to residents responses on survey –5 year approval - commendation

ACGME Survey - Problems Answer selections vague High threshold for compliance Small program effect Consequences of survey punitive Consequences based on survey results and not actual work hours data

Current Study Purpose – obtain general surgery residents views on work hours regulations and ACGME survey Design – 20 question survey ed to all Gen. Surg. Program Directors – 5/09 –9 questions – work hours –9 questions – ACGME survey –1 both, 1 demographic, comments Study period – 5/09 – 10/09 –Multiple reminders, pleas for participation Internet Survey – surveymonkey.com

Questions TABLE 1: Questions from Surgical Residents Views on the Accreditation Council for Graduate Medical Education (ACGME) Resident Survey. 1. How many times have you taken the ACGME Resident survey? 2. Did you have any difficulty understanding the questions as written in the ACGME Survey? 3. Did you discuss the ACGME Resident Survey with your residency Program Director or any of your faculty prior to completing the survey? 4. Were you, in any way, instructed how to answer questions on the ACGME Resident Survey? 5. Did you answer all questions on the ACGME Resident Survey truthfully? 6. Are you aware how your answers to the ACGME Resident Survey can effect your surgery program's accreditation? 7. If you indicated an ACGME duty hours requirement is "SOMETIMES" met, which of the following is true? 8. If an ACGME duty hours requirement is "SOMETIMES" met, how often are you NONCOMPLIANT with this requirement? 9. When answering the ACGME Survey, what percentage of residents in a program need to indicate a rule violation for this to be deemed significant by the ACGME? 10. Do you feel the ACGME Resident Survey accurately evaluated the hours that you work in your Residency Program?

Results Answers collected, analyzed - 11/09 Aggregate Level of training –Junior – PGY 1, 2 –Senior – PGY > 2 Statistical analysis – openepi.com –P value < 0.05, statistically significant

Results - Responses Resident Year of Training by Post Graduate Year (PGY) Number of Resident Responses Percent of Total PGY PGY PGY PGY PGY 5 or Greater Total – 13.1 % of U.S. General Surgery Residents 961 completed entire survey (99.6%) Junior Residents – 441 Senior Residents – 520

Results Number Times Survey Taken Question Junior Residents (% of Junior Residents) Senior Residents (% of Senior Residents) Total (%) P values How many times have you taken the ACGME Resident/Fellow Survey? 1 time = 254 (58) 2 times = 148 (34) 3 times = 25 (5) 4 times = 9 (2) 5 times = 1 (0.1) > 5 times = 4 (0.9) 1 time = 93 (17) 2 times = 95 (18) 3 times = 164 (31) 4 times = 90 (17) 5 times = 58 (10) > 5 times = 40 (7) 1 time = 347 (36) 2 times = 243 (25) 3 times = 189 (19) 4 times = 99 (10) 5 times = 59 (6) >5 times = 44 (4) <

Results Instruction on Survey Question Junior Residents (% of Junior Residents) Senior Residents (% of Senior Residents) Total (%)P values Discuss survey with PD or faculty Yes = 156 (35) No = 285 (65) Yes = 185 (36) No = 335 (64) Yes = 341 (35) No = 620 (65) Instructed on how to answer survey Yes = 84 (19) No = 357 (81) Yes = 82 (16) No = 438 (84) Yes = 166 (17) No = 795 (83)

Results – Survey Interpretation Question Junior Residents (% of Junior Residents) Senior Residents (% of Senior Residents) Total (%) P values Difficulty Understanding Questions Yes = 61 (14) No = 380 (86) Yes = 118 (23) No = 402 (77) Yes = 179 (19) No =782 (81) When requirement sometimes met, how often noncompliant In compliance = 121 (27) In violation = 202 (46) Neither compliant nor in violation = 118 (27) In compliance = 156 (30) In violation = 236 (45) Neither compliant nor in violation = 128 (25) In compliance = 280 (29) In violation = 438 (45) Neither compliant nor in violation = 246 (26) Meaning of sometimes times/year = 212 (48) times/year = 117 (27) times/year = 41 (9) times/year = 27 (6) times/year = 11 (3) > 25 times/year = 33 (7) times/year = 288 (55) times/year = 109 (20) times/year = 49 (10) times/year = 25 (5) times/year = 10 (2) > 25 times/year = 39 (8) times/year = 500 (52) times/year = 226 (24) times/year =90 (10) times/year = 52 (5) times/year = 21(2) > 25 times/year = 72 (7)

Results Effect on Program Question Junior Residents (% of Junior Residents) Senior Residents (% of Senior Residents) Total (%) P values Aware of effect on program accreditation Yes = 337 (76) No = 103 (24) Yes = 399 (77) No = 121 (23) Yes = 736 (77) No = 224 (23) What percent necessary to be deemed significant by ACGME 5% = 131 (30) 10% = 131 (30) 15% = 37 (18) 20% = 68 (15) 25% = 52 (12) 50% = 22 (5) 5% = 161 (31) 10% = 117 (23) 15% = 51 (10) 20% = 97 (18) 25% = 43 (8) 50% = 51 (10) 5% = 292 (30) 10% = 248 (26) 15% = 88 (9) 20% = 165 (17) 25% = 95 (10) 50% = 73 (8)

Results Honesty/Accuracy Question Junior Residents (% of Junior Residents) Senior Residents (% of Senior Residents) Total (%) P values Answer questions truthfully Yes = 382 (87) No = 59 (13) Yes = 446 (86) No = 74 (14) Yes = 828 (86) No = 133 (14) ACGME Survey accurately evaluated hours that you work Yes = 287 (65) No = 154 (35) Yes = 319 (61) No = 201 (39) Yes = 606 (63) No =355 (37)

Conclusions Evaluation mechanism in which –52% do not understand questions –14% did not answer truthfully –37% do not feel is accurate Not best tool to evaluate work hours compliance in residency programs Consequences significant Open letter to GME community 10/28/09 – Nasca, T.

Thank you !! Program Directors Residents