Managing the ACGME Survey OPDO Meeting November 11, 2011 I’ve been asked to share some tips on how to manage the ACGME Survey, which is admittedly not one of the flashier topics for today… but there are definitely some common obstacles we face as our residents take this survey, so hopefully it is helpful to talk through some of these challenges. Marita S. Teng, MD Program Director Department of Otolaryngology Mount Sinai School of Medicine New York, NY

ACGME Survey – What is it? Online survey, 34 questions Administered annually for residents and fellows Timing: mid-January thru early June Requires 70% completion rate Aggregate program-level data provided to PD and DIO if 70% response rate reached Site visitors and RRC have data access regardless of response rate

ACGME Survey – What is it? Topics covered Duty Hours Faculty Evaluation Systems Educational Content Resources Overall Experience 6 major topics covered by survey

Data from Survey From the ACGME website “Data from the survey are intended as a diagnostic tool to focus the field staff’s questions during the on-site visit… [and] to verify and clarify the information from them. Field staff will probe and clarify any grey shaded areas, and pay particular attention to the duty hour items and to items with a substantial number of non-compliant responses.” If residents indicate problem areas corrected, site visitor will document. If questions/phrases misunderstood by residents, site visitor will document. What is the data from the survey used for? Sounds pretty innocuous.

When to Worry? “Substantial number of noncompliant responses” 20% is a common institutional standard Internal systematic review and action plan required RRC may send warning letters to programs for poor results Accreditation cycle may be shortened, effectively triggering an immediate site visit They will “pay particular attention” to DH questions and non-compliant responses. “Substantial number” is not defined, but… And when survey results are VERY concerning,…

Problems specific to us Small programs One or two disgruntled residents can significantly alter the data Confidentiality is affected Interns Data often reflect their experiences on services other than Otolaryngology, and are hard to pinpoint because of varying rotation schedules and anonymity of survey We are all small programs, so…. Confidentiality is a theme that comes up a couple times on this survey. (evals but also dealing with problems) – and things that are supposed to be confidential are harder to keep confidential when there are so few people (faculty and residents) And PGY1’s are OURS even though they are not on service for the majority of the year. But no “excuses” are accepted!

“Safe Surveying” Survey instructions sent out only after a intro/Q&A session Explain purpose of survey, clarify questions Survey form available on ACGME website – go item by item This is not the forum for complaints Consider NOT having the PD conduct the session Associate PD? Chief resident(s)? Program coordinator? For all these reasons, we should all practice “safe surveying”! PD gets notified of survey first, and is responsible for sending out the instructions, which gives you the opportunity to…

“Safe Surveying” Remind the residents about certain items: Duty hours Confidentiality of your evaluation system Availability/accessibility of Goals & Objectives Resources & systems for communication/support education committee meetings faculty mentors chief or senior residents institutional ombudsman Sleep deprivation education SAFER curriculum (ppt) Most dangerous issues are with DH – if residents are not familiar with the rules. It also pays to remind them about certain existing things that they may have forgotten about… Sleep Alertness & Fatigue Education in Residency – 2MB ppt I can send anyone who is interested – created in 2006 by American Academy of Sleep Medicine

Otolaryngology Problem Areas We are better than most! Focus on areas of around 5% or greater national rates of non-compliance The good news is, we are better than most! With the proper preparation for the survey, the vast majority of ENT programs will not be “at risk” for penalties from poor survey results. There are a few areas where nationally, we hover around 5% or greater of noncompliance, and I’ll focus on those areas, so we know where we need to direct our efforts in general. I’m going to go through those 6 sections of the survey, section by section, and share only the “problem” questions for us as a whole.

Duty Hours Seven Questions 80-hour week (over 4 wks) 1 day off in 7 (over 4 wks) No more than Q3 call (over 4 wks) 10 hours off between duty periods 24 + 6 continuous hours in-house Home call does not preclude “rest and reasonable personal time” Called into hospital from home  counts towards 80hrs There are 7 questions pertaining to DH, which accounts for about 20% of the entire survey. Plus they state that special attention is paid to this section, so it goes without saying that our residents need to abide by and understand all the rules. It is worth reviewing on a regular basis, including right before the survey is given.

Compiled Otolaryngology Data: Duty Hours Never Rarely Sometimes Very Often Extremely Often How often did you break the rule that there should be a 10-hour time period provided between all daily duty periods and after in-house call? 71.9% 22.6% 4.9% 0.7% 0% Looking at the aggregate data for ENT programs nationwide, the only DH question that demonstrates 5% noncompliance is the 10 hrs off rule. DH question guide is a good resource. That’s really all I’m going to say about the DH section, because it’s really the most concrete portion of the survey, and there’s not much to add. Duty hour question guide (PDF) for residents on ACGME website

Compiled Otolaryngology Data: Faculty Extremely Very Somewhat Slightly Not at all How sufficient is the instruction you receive from faculty and staff in your program? 59.7% 34.3% 5.3% 0.4% 0.1% Thinking about the faculty overall, how interested are they in your residency education? 64.1% 28.6% 5.8% 1.6% 0% Thinking about the faculty overall, how effective are they in creating an environment of scholarship and inquiry? 61.4% 30.4% 6.8% 1.4% Moving onto the faculty section, again with aggregate national data, about 6-8% of residents gave noncompliant responses to 3 of the questions… overall this isn’t bad, but leaves some room for improvement.

Faculty Issues Consider faculty workshops Resources Mentorship Giving feedback Teaching techniques Resources GME office Dean’s office Faculty Development Programs Faculty issues are important, but obviously it is not something you can coach in terms of survey answers. This cartoon reminds me of how things go in academic medicine these days… there are so many demands on our time, and it seems whenever we try to change the system or streamline something, it requires the formation of some committee or subcommittee, which of course ironically just takes up more time. Teaching and mentorship almost always get de-prioritized. It is nice to periodically re-focus our efforts, and there are resources for that.

Compiled Otolaryngology Data: Evaluation Extremely Very Somewhat Slightly Not at all How satisfied are you that your program treats your evaluations of faculty members confidentially? 62.5% 28.0% 5.5% 1.3% 0.4% How satisfied are you with the way your program uses the evaluations that residents provide to improve the program? 55.0% 30.4% 10.5% 2.4% 0.6% Overall, how satisfied are you with the written or electronic feedback you receive after you complete a rotation or major assignment? 50.4% 35.8% 10.8% 2.3% 0.7% The 1st question demonstrates that there is some doubt about the confidentiality of the residents’ evaluation of the attendings – this may be largely due to the small-program phenomenon that I mentioned before. The 2nd and 3rd responses are concerning from the standpoint that some of the residents don’t feel that their feedback- either given or received - is meaningful.

Evaluation issues Confidentiality is compromised by nature of being a small program Consider pooling attending or rotation evaluations, then distribute at the end of each year Evaluations need to be meaningful Shortening the form Requiring written comments Issues identified by residents should be followed up; improvements should be documented Evaluation systems in small programs are technically confidential, but in reality, on a small service, it becomes pretty transparent. Consider pooling! Documenting changes and improvements in the program are helpful not only for ongoing efforts, but you will be thankful when the RRC site visit occurs!

Compiled Otolaryngology Data: Educational Content Extremely Very Sometimes/ somewhat Slightly Not at all How satisfied are you with the opportunities your program provides for you to participate in research or scholarly activities? 59.6% 31.0% 7.9% 1.3% 0.2% In your opinion, how often do your rotations provide an appropriate balance between your residency education and other clinical demands? 50.7% 37.8% 10.2% 1.2% 0% How often has your clinical education been compromised by excessive service obligations? 0.7% 1.9% 15.8% 35.3% 46.3% These last 2 are our biggest problem areas; that holds true for all programs obviously, not just ENT, and there really are no easy solutions.

Educational Content Issues Time for research How protected is the time? Balance between education and service Physician extenders – “non-educational” work Continually re-evaluate quality of didactics Faculty mentorship & teaching

Compiled Otolaryngology Data: Resources Extremely Very Sometimes/somewhat Slightly/rarely Not at all How often do you work in interdisciplinary teams to care for patients? 64.8% 28.9% 5.6% 0.7% 0.1% How satisfied are you with your program’s process to deal confidentially with problems or concerns residents might have? 58.8% 30.0% 8.5% 2.2% 0.5% To what extent does your program provide an environment where residents can raise problems or concerns without fear of intimidation or retaliation? 64.9% 25.7% 6.8% 2.1% 0.4%

Resource Issues Interdisciplinary teams Tumor Board? Case conferences? Combined Grand Rounds? QM or QI teams? Dealing with problems confidentially Institutional resources in the GME office Dealing with problems without fear of intimidation or retaliation Resident-only meetings

Overall Experience Which of the following best summarizes your opinion of your residency program?

Take-Home Points Explain and prepare residents Increase communication Purpose of the survey Specific questions Can re-administer survey internally Increase communication Regular meetings with residents Regular meetings with faculty Lots of channels Increase overall morale We are in good shape! Take as opportunity to improve program

Thank You Mount Sinai School of Medicine GME office Ben Malkin, MD Scott Barnett, MD – Associate Dean for GME Paul Johnson Ben Malkin, MD Associate Program Director, ENT, MSSM