1. Is The ACGME Resident/Fellow Survey a Valid Tool to Assess General Surgery Residency Programs Compliance with Work Hours Regulations? Robert P. Sticca, MD, FACS Jay M. MacGregor, MD Randolph E. Szlabick, MD, FACS Department of Surgery University of North Dakota SMHS

2. Background 2003 – ACGME work hours regulations implemented 2004 – ACGME Residents/Fellows Survey 2008 – Special Message from CEO ACGME (Thomas Nasca, MD) –Standard Approach to Programs Across All Specialties with Potential Duty Hours Violations Identified in the Resident Survey

3. ACGME Special Message The Monitoring Committee has accepted the Resident Survey as a reliable, sensitive and reasonably specific tool The Monitoring Committee has set reasonable thresholds for categorization of a program as having Potential Duty Hours Violations (PDHV) based on the Resident Survey The validity of the Resident Survey has been demonstrated though significant correlations with site visit findings to warrant added scrutiny for programs meeting the thresholds of potential noncompliance found in the group targeted for follow-up. There is a significant correlation between resident reported violations of duty hours with deficiencies in other important areas of the learning environment (correlation 0.51, p<0.0001) Nasca, T. www.ACGME.org

4. ACGME Algorithm for Noncompliant Programs Nasca, T. www.ACGME.org

5. UND Experience 2005 – RRC site visit – additional categorical resident requested –5 year approval, no deficiencies –Additional resident denied, insufficient educational justification 2007 – RRC site visit – additional categorical resident requested –5 year approval, no deficiencies –Additional resident approved 2009 – RRC site visit – mandatory due to residents responses on survey –5 year approval - commendation

6. ACGME Survey - Problems Answer selections vague High threshold for compliance Small program effect Consequences of survey punitive Consequences based on survey results and not actual work hours data

7. Current Study Purpose – obtain general surgery residents views on work hours regulations and ACGME survey Design – 20 question survey emailed to all Gen. Surg. Program Directors – 5/09 –9 questions – work hours –9 questions – ACGME survey –1 both, 1 demographic, comments Study period – 5/09 – 10/09 –Multiple reminders, pleas for participation Internet Survey – surveymonkey.com

8. Questions TABLE 1: Questions from Surgical Residents Views on the Accreditation Council for Graduate Medical Education (ACGME) Resident Survey. 1. How many times have you taken the ACGME Resident survey? 2. Did you have any difficulty understanding the questions as written in the ACGME Survey? 3. Did you discuss the ACGME Resident Survey with your residency Program Director or any of your faculty prior to completing the survey? 4. Were you, in any way, instructed how to answer questions on the ACGME Resident Survey? 5. Did you answer all questions on the ACGME Resident Survey truthfully? 6. Are you aware how your answers to the ACGME Resident Survey can effect your surgery program's accreditation? 7. If you indicated an ACGME duty hours requirement is "SOMETIMES" met, which of the following is true? 8. If an ACGME duty hours requirement is "SOMETIMES" met, how often are you NONCOMPLIANT with this requirement? 9. When answering the ACGME Survey, what percentage of residents in a program need to indicate a rule violation for this to be deemed significant by the ACGME? 10. Do you feel the ACGME Resident Survey accurately evaluated the hours that you work in your Residency Program?

9. Results Answers collected, analyzed - 11/09 Aggregate Level of training –Junior – PGY 1, 2 –Senior – PGY > 2 Statistical analysis – openepi.com –P value < 0.05, statistically significant

10. Results - Responses Resident Year of Training by Post Graduate Year (PGY) Number of Resident Responses Percent of Total PGY 124625.6 PGY 219520.3 PGY 318118.8 PGY 416016.7 PGY 5 or Greater17918.6 Total961100 965 – 13.1 % of U.S. General Surgery Residents 961 completed entire survey (99.6%) Junior Residents – 441 Senior Residents – 520

11. Results Number Times Survey Taken Question Junior Residents (% of Junior Residents) Senior Residents (% of Senior Residents) Total (%) P values How many times have you taken the ACGME Resident/Fellow Survey? 1 time = 254 (58) 2 times = 148 (34) 3 times = 25 (5) 4 times = 9 (2) 5 times = 1 (0.1) > 5 times = 4 (0.9) 1 time = 93 (17) 2 times = 95 (18) 3 times = 164 (31) 4 times = 90 (17) 5 times = 58 (10) > 5 times = 40 (7) 1 time = 347 (36) 2 times = 243 (25) 3 times = 189 (19) 4 times = 99 (10) 5 times = 59 (6) >5 times = 44 (4) <0.0000001

12. Results Instruction on Survey Question Junior Residents (% of Junior Residents) Senior Residents (% of Senior Residents) Total (%)P values Discuss survey with PD or faculty Yes = 156 (35) No = 285 (65) Yes = 185 (36) No = 335 (64) Yes = 341 (35) No = 620 (65) 0.8881 Instructed on how to answer survey Yes = 84 (19) No = 357 (81) Yes = 82 (16) No = 438 (84) Yes = 166 (17) No = 795 (83) 0.2242

13. Results – Survey Interpretation Question Junior Residents (% of Junior Residents) Senior Residents (% of Senior Residents) Total (%) P values Difficulty Understanding Questions Yes = 61 (14) No = 380 (86) Yes = 118 (23) No = 402 (77) Yes = 179 (19) No =782 (81) 0.0002662 When requirement sometimes met, how often noncompliant In compliance = 121 (27) In violation = 202 (46) Neither compliant nor in violation = 118 (27) In compliance = 156 (30) In violation = 236 (45) Neither compliant nor in violation = 128 (25) In compliance = 280 (29) In violation = 438 (45) Neither compliant nor in violation = 246 (26) 0.4817 Meaning of sometimes 1 - 5 times/year = 212 (48) 6 - 10 times/year = 117 (27) 11 - 15 times/year = 41 (9) 16 - 20 times/year = 27 (6) 21 -25 times/year = 11 (3) > 25 times/year = 33 (7) 1 - 5 times/year = 288 (55) 6 - 10 times/year = 109 (20) 11 - 15 times/year = 49 (10) 16 - 20 times/year = 25 (5) 21 -25 times/year = 10 (2) > 25 times/year = 39 (8) 1 - 5 times/year = 500 (52) 6 - 10 times/year = 226 (24) 11 - 15 times/year =90 (10) 16 - 20 times/year = 52 (5) 21 -25 times/year = 21(2) > 25 times/year = 72 (7) 0.2852

14. Results Effect on Program Question Junior Residents (% of Junior Residents) Senior Residents (% of Senior Residents) Total (%) P values Aware of effect on program accreditation Yes = 337 (76) No = 103 (24) Yes = 399 (77) No = 121 (23) Yes = 736 (77) No = 224 (23) 0.9081 What percent necessary to be deemed significant by ACGME 5% = 131 (30) 10% = 131 (30) 15% = 37 (18) 20% = 68 (15) 25% = 52 (12) 50% = 22 (5) 5% = 161 (31) 10% = 117 (23) 15% = 51 (10) 20% = 97 (18) 25% = 43 (8) 50% = 51 (10) 5% = 292 (30) 10% = 248 (26) 15% = 88 (9) 20% = 165 (17) 25% = 95 (10) 50% = 73 (8) 0.005127

15. Results Honesty/Accuracy Question Junior Residents (% of Junior Residents) Senior Residents (% of Senior Residents) Total (%) P values Answer questions truthfully Yes = 382 (87) No = 59 (13) Yes = 446 (86) No = 74 (14) Yes = 828 (86) No = 133 (14) 0.8856 ACGME Survey accurately evaluated hours that you work Yes = 287 (65) No = 154 (35) Yes = 319 (61) No = 201 (39) Yes = 606 (63) No =355 (37) 0.2231

16. Conclusions Evaluation mechanism in which –52% do not understand questions –14% did not answer truthfully –37% do not feel is accurate Not best tool to evaluate work hours compliance in residency programs Consequences significant Open letter to GME community 10/28/09 –www.acgme.org Nasca, T. www.ACGME.org

17. Thank you !! Program Directors Residents