1. Study question: A common difficulty of psychiatry residency training programs is determining how to address and quantify culture or administration problems. Chief Residents are often in the position to mediate discussion and formulate solutions, but little information exists on experiences of how to accomplish this task. Designing a Survey to Assess the Health and Culture of a Residency Program Rebecca Gladding, M.D., Dana Levy, M.D., Randall Espinoza, M.D., Ashley Dorin, M.D. and Peter Whybrow, M.D. UCLA Department of Psychiatry and Biobehavioral Sciences Methods: Beginning with data from our annual retreat and a series of focus groups, we developed a 108-question survey for residents that assessed the strengths, weaknesses and areas for improvement in the following realms: goals and philosophy of the program, professional development, well-being, administration, call, feedback and supervision and PGY specific issues. We sent the survey link out to residents via email more than five times to achieve a response rate of 70%. We also developed a companion study for faculty and administrators mostly related to feedback to ensure that perspectives from all sides were incorporated. Results: The resident response rate to complete entire survey was 56%, though 67% started the survey. PGY1-3 initial responses averaged 78%, which dropped to 71% for completion. After the first week, the survey had to be divided into two parts based on feedback from residents that it was “too long” and taking “too much time” to complete. Residents also negatively commented on having too many required questions on each page of the Survey Monkey form. For faculty, 37 started the survey and 27 completed it, totaling 73% of those who originally responded. Total faculty response was low given that more than 100 faculty were invited to respond to the survey; however, those who did respond work closely with residents. Numerical/categorical data yielded helpful information to validate or reject certain assertions about the program, such as assessing the overall health of a program: Conclusions: Free-form questions can highlight issues you do not know exist or themes that run across various groups (e.g., residents and faculty), especially when prompted by, or combined with, categorical questions Categorical questions can yield helpful overall state of the residency information and test assumptions/myths of the program in an neutral/non-confrontational way Survey Design: Must be anonymous when highly sensitive topics are being assessed Shorter, focused surveys are better received by busy participants Need to survey all stakeholders as appropriate (e.g., faculty, nursing) Define terms clearly to avoid confusion or bias (ex: what does “excellent” vs. “great” vs. “good” vs. “fair” mean) State questions neutrally and give respondents an opportunity to disagree with the statement or question; avoid leading questions when possible Avoid too many free-form questions, as they take more time to complete and require more thought per response Do not require all questions be answered; rather, focus on requiring responses only to the most important ones. Sample Combined Categorical and Free-Form Questions Attendings: 1) Do you feel like the culture is supportive of giving feedback to residents and backing up attendings when they give that feedback? 2) In what circumstances, or for what reasons, would you not give a resident feedback? Reference: Fraenkel JR, Wallen, NE (2003). How to design and evaluate research in education, 5th ed. San Francisco, CA : McGraw-Hill, page 395. Sample Categorical Questions Residents: How often do you receive positive feedback? Helpful/constructive feedback? Attendings: How often do you give residents positive feedback on their work? Constructive feedback? Freeform responses highlighted issues that had never been overtly articulated, but were problematic undercurrents within the residency. With regard to feedback, when asked “what are the major issues you faced as an intern?” one person commented: “Have attendings tell us what their expectations for a call presentation [are].” In a different section regarding possibly learning how to give feedback, a resident responded: “More useful to have a culture where such feedback is expected and encouraged.” These sentiments were mirrored in faculty responses, as shown below. Combining categorical and free-form responses yielded the most informative results, both within one question and across similar questions. For example, asking categorical questions about feedback with an option to provide free-form text helped identify a significant issue in the residency: lack of clear expectations regarding giving feedback within the culture. Attendings: “The culture does not support giving feedback as there is no structural framework to do so regularly, and so when it is done, it seems extraordinary.”