1. Managing the ACGME Survey OPDO Meeting November 11, 2011
I’ve been asked to share some tips on how to manage the ACGME Survey, which is admittedly not one of the flashier topics for today… but there are definitely some common obstacles we face as our residents take this survey, so hopefully it is helpful to talk through some of these challenges.
Marita S. Teng, MD
Program Director
Department of Otolaryngology
Mount Sinai School of Medicine
New York, NY

2. ACGME Survey – What is it?
Online survey, 34 questions
Administered annually for residents and fellows
Timing: mid-January thru early June
Requires 70% completion rate
Aggregate program-level data provided to PD and DIO if 70% response rate reached
Site visitors and RRC have data access regardless of response rate

3. ACGME Survey – What is it?
Topics covered
Duty Hours
Faculty
Evaluation Systems
Educational Content
Resources
Overall Experience
6 major topics covered by survey

4. Data from Survey From the ACGME website
“Data from the survey are intended as a diagnostic tool to focus the field staff’s questions during the on-site visit… [and] to verify and clarify the information from them. Field staff will probe and clarify any grey shaded areas, and pay particular attention to the duty hour items and to items with a substantial number of non-compliant responses.”
If residents indicate problem areas corrected, site visitor will document.
If questions/phrases misunderstood by residents, site visitor will document.
What is the data from the survey used for? Sounds pretty innocuous.

5. When to Worry? “Substantial number of noncompliant responses”
20% is a common institutional standard
Internal systematic review and action plan required
RRC may send warning letters to programs for poor results
Accreditation cycle may be shortened, effectively triggering an immediate site visit
They will “pay particular attention” to DH questions and non-compliant responses. “Substantial number” is not defined, but… And when survey results are VERY concerning,…

6. Problems specific to us
Small programs
One or two disgruntled residents can significantly alter the data
Confidentiality is affected
Interns
Data often reflect their experiences on services other than Otolaryngology, and are hard to pinpoint because of varying rotation schedules and anonymity of survey
We are all small programs, so….
Confidentiality is a theme that comes up a couple times on this survey. (evals but also dealing with problems) – and things that are supposed to be confidential are harder to keep confidential when there are so few people (faculty and residents)
And PGY1’s are OURS even though they are not on service for the majority of the year.
But no “excuses” are accepted!

7. “Safe Surveying”
Survey instructions sent out only after a intro/Q&A session
Explain purpose of survey, clarify questions
Survey form available on ACGME website – go item by item
This is not the forum for complaints
Consider NOT having the PD conduct the session
Associate PD?
Chief resident(s)?
Program coordinator?
For all these reasons, we should all practice “safe surveying”! PD gets notified of survey first, and is responsible for sending out the instructions, which gives you the opportunity to…

8. “Safe Surveying” Remind the residents about certain items: Duty hours
Confidentiality of your evaluation system
Availability/accessibility of Goals & Objectives
Resources & systems for communication/support
education committee meetings
faculty mentors
chief or senior residents
institutional ombudsman
Sleep deprivation education
SAFER curriculum (ppt)
Most dangerous issues are with DH – if residents are not familiar with the rules. It also pays to remind them about certain existing things that they may have forgotten about…
Sleep Alertness & Fatigue Education in Residency – 2MB ppt I can send anyone who is interested – created in 2006 by American Academy of Sleep Medicine

9. Otolaryngology Problem Areas
We are better than most!
Focus on areas of around 5% or greater national rates of non-compliance
The good news is, we are better than most! With the proper preparation for the survey, the vast majority of ENT programs will not be “at risk” for penalties from poor survey results. There are a few areas where nationally, we hover around 5% or greater of noncompliance, and I’ll focus on those areas, so we know where we need to direct our efforts in general. I’m going to go through those 6 sections of the survey, section by section, and share only the “problem” questions for us as a whole.

10. Duty Hours Seven Questions
80-hour week (over 4 wks)
1 day off in 7 (over 4 wks)
No more than Q3 call (over 4 wks)
10 hours off between duty periods
continuous hours in-house
Home call does not preclude “rest and reasonable personal time”
Called into hospital from home  counts towards 80hrs
There are 7 questions pertaining to DH, which accounts for about 20% of the entire survey. Plus they state that special attention is paid to this section, so it goes without saying that our residents need to abide by and understand all the rules. It is worth reviewing on a regular basis, including right before the survey is given.

11. Compiled Otolaryngology Data: Duty Hours
Never
Rarely
Sometimes
Very
Often
Extremely Often
How often did you break the rule that there should be a 10-hour time period provided between all daily duty periods and after in-house call?
71.9%
22.6%
4.9%
0.7% 0%
Looking at the aggregate data for ENT programs nationwide, the only DH question that demonstrates 5% noncompliance is the 10 hrs off rule. DH question guide is a good resource. That’s really all I’m going to say about the DH section, because it’s really the most concrete portion of the survey, and there’s not much to add.
Duty hour question guide (PDF)
for residents on ACGME website

12. Compiled Otolaryngology Data: Faculty
Extremely
Very
Somewhat
Slightly
Not at all
How sufficient is the instruction you receive from faculty and staff in your program?
59.7%
34.3%
5.3%
0.4%
0.1%
Thinking about the faculty overall, how interested are they in your residency education?
64.1%
28.6%
5.8%
1.6% 0%
Thinking about the faculty overall, how effective are they in creating an environment of scholarship and inquiry?
61.4%
30.4%
6.8%
1.4%
Moving onto the faculty section, again with aggregate national data, about 6-8% of residents gave noncompliant responses to 3 of the questions… overall this isn’t bad, but leaves some room for improvement.

13. Faculty Issues Consider faculty workshops Resources Mentorship
Giving feedback
Teaching techniques
Resources
GME office
Dean’s office
Faculty Development Programs
Faculty issues are important, but obviously it is not something you can coach in terms of survey answers. This cartoon reminds me of how things go in academic medicine these days… there are so many demands on our time, and it seems whenever we try to change the system or streamline something, it requires the formation of some committee or subcommittee, which of course ironically just takes up more time. Teaching and mentorship almost always get de-prioritized. It is nice to periodically re-focus our efforts, and there are resources for that.

14. Compiled Otolaryngology Data: Evaluation
Extremely
Very
Somewhat
Slightly
Not at all
How satisfied are you that your program treats your evaluations of faculty members confidentially?
62.5%
28.0%
5.5%
1.3%
0.4%
How satisfied are you with the way your program uses the evaluations that residents provide to improve the program?
55.0%
30.4%
10.5%
2.4%
0.6%
Overall, how satisfied are you with the written or electronic feedback you receive after you complete a rotation or major assignment?
50.4%
35.8%
10.8%
2.3%
0.7%
The 1st question demonstrates that there is some doubt about the confidentiality of the residents’ evaluation of the attendings – this may be largely due to the small-program phenomenon that I mentioned before. The 2nd and 3rd responses are concerning from the standpoint that some of the residents don’t feel that their feedback- either given or received - is meaningful.

15. Evaluation issues
Confidentiality is compromised by nature of being a small program
Consider pooling attending or rotation evaluations, then distribute at the end of each year
Evaluations need to be meaningful
Shortening the form
Requiring written comments
Issues identified by residents should be followed up; improvements should be documented
Evaluation systems in small programs are technically confidential, but in reality, on a small service, it becomes pretty transparent. Consider pooling!
Documenting changes and improvements in the program are helpful not only for ongoing efforts, but you will be thankful when the RRC site visit occurs!

16. Compiled Otolaryngology Data: Educational Content
Extremely
Very
Sometimes/
somewhat
Slightly
Not at all
How satisfied are you with the opportunities your program provides for you to participate in research or scholarly activities?
59.6%
31.0%
7.9%
1.3%
0.2%
In your opinion, how often do your rotations provide an appropriate balance between your residency education and other clinical demands?
50.7%
37.8%
10.2%
1.2% 0%
How often has your clinical education been compromised by excessive service obligations?
0.7%
1.9%
15.8%
35.3%
46.3%
These last 2 are our biggest problem areas; that holds true for all programs obviously, not just ENT, and there really are no easy solutions.

17. Educational Content Issues
Time for research
How protected is the time?
Balance between education and service
Physician extenders – “non-educational” work
Continually re-evaluate quality of didactics
Faculty mentorship & teaching

18. Compiled Otolaryngology Data: Resources
Extremely
Very
Sometimes/somewhat
Slightly/rarely
Not at all
How often do you work in interdisciplinary teams to care for patients?
64.8%
28.9%
5.6%
0.7%
0.1%
How satisfied are you with your program’s process to deal confidentially with problems or concerns residents might have?
58.8%
30.0%
8.5%
2.2%
0.5%
To what extent does your program provide an environment where residents can raise problems or concerns without fear of intimidation or retaliation?
64.9%
25.7%
6.8%
2.1%
0.4%

19. Resource Issues Interdisciplinary teams
Tumor Board?
Case conferences?
Combined Grand Rounds?
QM or QI teams?
Dealing with problems confidentially
Institutional resources in the GME office
Dealing with problems without fear of intimidation or retaliation
Resident-only meetings

20. Overall Experience
Which of the following best summarizes your opinion of your residency program?

21. Take-Home Points Explain and prepare residents Increase communication
Purpose of the survey
Specific questions
Can re-administer survey internally
Increase communication
Regular meetings with residents
Regular meetings with faculty
Lots of channels
Increase overall morale
We are in good shape!
Take as opportunity to improve program

22. Thank You Mount Sinai School of Medicine GME office Ben Malkin, MD
Scott Barnett, MD – Associate Dean for GME
Paul Johnson
Ben Malkin, MD
Associate Program Director, ENT, MSSM