1. 360 DEGREE FEEDBACK: A NOVEL FORMAT FOR A PROGRAM EVALUATION COMMITTEE IN AN ACADEMIC EMERGENCY MEDICINE RESIDENCY PROGRAM Holly Caretta-Weyer, MD; Morgan Wilbanks, MD; Bryce Snow, MD; Aaron Kraut, MD; Ciara Barclay-Buchanan, MD; Mary Westergaard, MD University of Wisconsin Department of Emergency Medicine INTRODUCTION PURPOSE AND EDUCATIONAL OBJECTIVES The ACGME mandates that residency programs maintain a program evaluation committee (PEC), which evaluates the educational activities of the program annually from the vantage point of resident and faculty stakeholders. One of the key functions of the PEC is to ensure the quality of the program across its educational offerings. PECs are historically structured to capture observations and input from faculty and resident stakeholders; however, feedback from non-traditional parties can provide further insight into the educational offerings of a program. ABSTRACT BACKGROUND: The ACGME mandates that residency programs maintain a program evaluation committee (PEC) which evaluates the educational activities of the program annually from the vantage point of resident and faculty stakeholders. While the traditional PEC structure does not include input from administrative, nursing, or ancillary staff, these individuals often have useful feedback. We sought to enrich our program evaluation process by eliciting 360 degree reviews of educational experiences from emergency medicine residents, off-service residents and faculty, nursing staff, administrators, and other relevant stakeholders in monthly, rotation-specific reviews. To our knowledge, this represents a novel approach to the PEC and provides the opportunity to explore untapped resources for improving educational experiences. EDUCATIONAL OBJECTIVES: 1.Initiate comprehensive, systematic evaluation of program educational experiences in line with function of PEC 2.Collect feedback from key stakeholders, including non-traditional sources, in order to identify actionable, high- yield recommendations for improvement 3.Establish longitudinal tracking of key recommendations to ensure implementation of meaningful change CURRICULAR DESIGN: We implemented a comprehensive multi-source review system in order to expand and accelerate our program evaluation process. Each month, five reviewers (chief resident, resident, program leader, core faculty, administrator) review a documentation packet and conduct interviews of key stakeholders (including non-traditional sources) using reviewer-specific interview tools. The findings are presented for residency-wide discussion at monthly meetings and then integrated into a final document highlighting strengths, areas of concern, and proposed improvements. This document is shared with the rotation director for feedback before negotiating final recommendations, which are tracked for successful implementation on a quarterly basis by the PEC. IMPACT: The 360 degree review process uncovers significant opportunities for improvement that are missed by traditional reviews. The negotiation process and formalized recommendations improve accountability on the part of rotation directors. Lastly, residents demonstrate improved understanding and engagement in program evaluation and improvement processes. We sought to enrich our program evaluation process by eliciting 360 degree reviews of educational experiences from emergency medicine residents, off-service residents and faculty, nursing staff, administrators, and other relevant stakeholders in monthly, rotation-specific reviews. To our knowledge, this represents a novel approach to the PEC and provides the opportunity to explore untapped resources for improving educational experiences. The objectives of this process were as follows: 1.Initiate comprehensive, systematic evaluation of program educational experiences in line with function of PEC 2.Collect feedback from key stakeholders, including non-traditional sources, in order to identify actionable, high-yield recommendations for improvement 3.Establish longitudinal tracking of key recommendations to ensure implementation of meaningful change

2. 360 DEGREE FEEDBACK: A NOVEL FORMAT FOR A PROGRAM EVALUATION COMMITTEE IN AN ACADEMIC EMERGENCY MEDICINE RESIDENCY PROGRAM Holly Caretta-Weyer, MD; Morgan Wilbanks, MD; Bryce Snow, MD; Aaron Kraut, MD; Ciara Barclay-Buchanan, MD; Mary Westergaard, MD University of Wisconsin Department of Emergency Medicine CURRICULAR DESIGN In order to expand and accelerate our program evaluation process, we implemented a comprehensive multi-source review system. Residency rotations are reviewed on an 18-month cycle with a new rotation reviewed each month. Each month, five reviewers (chief resident, emergency medicine resident, program director or assistant program director, core faculty member, and program administrator) review a documentation packet and conduct interviews of key stakeholders, including non-traditional sources, using reviewer-specific interview tools. The findings are presented for residency-wide discussion at monthly meetings and then integrated into a final document highlighting strengths, areas of concern, and proposed improvements. This document is shared with the rotation director for feedback prior to negotiating final recommendations, which are tracked for successful implementation on a quarterly basis by the PEC. Figure 1: Administrative Reviewer Worksheet

3. 360 DEGREE FEEDBACK: A NOVEL FORMAT FOR A PROGRAM EVALUATION COMMITTEE IN AN ACADEMIC EMERGENCY MEDICINE RESIDENCY PROGRAM Holly Caretta-Weyer, MD; Morgan Wilbanks, MD; Bryce Snow, MD; Aaron Kraut, MD; Ciara Barclay-Buchanan, MD; Mary Westergaard, MD University of Wisconsin Department of Emergency Medicine Figure 3: Example of Focused Review Summary Template Figure 2: Resident Reviewer Worksheet

4. 360 DEGREE FEEDBACK: A NOVEL FORMAT FOR A PROGRAM EVALUATION COMMITTEE IN AN ACADEMIC EMERGENCY MEDICINE RESIDENCY PROGRAM Holly Caretta-Weyer, MD; Morgan Wilbanks, MD; Bryce Snow, MD; Aaron Kraut, MD; Ciara Barclay-Buchanan, MD; Mary Westergaard, MD University of Wisconsin Department of Emergency Medicine The 360 degree review process has uncovered significant opportunities for improvement that are missed by traditional reviews and has led to meaningful changes for resident education and patient care. Examples include, but are not limited to: - Introduction of a formal orientation curriculum for off-service rotations - End-of-day feedback cards for emergency medicine residents on off-service rotations - Protected obstetric ultrasound time on the OBGYN rotation The negotiation process and formalized recommendations have improved accountability on the part of rotation directors. This feedback model has already had a profound impact on resident satisfaction with rotations, educational experiences, improved understanding and engagement with the program evaluation and improvement process, and patient care. Results of resident feedback from the biannual review data: - The initial review found residents nearly uniformly view the education committee systematic review process to be “very effective.” - Several residents noted that the deep dive evaluation process generates actionable items that are then taken to the rotation leadership of other departments by our residency program leadership. This has enacted rapid,meaningful, and tangible change in rotations reviewed thus far. - One of the key strengths identified in the biannual resident reviews was that residents, faculty, and administrative staff gather data independently and meet to then synthesize the information to produce strategies for improvement. The varying viewpoints and amount of data reviewed makes the reviews “much more robust.” IMPACT The novel use of the acquisition of multi-source feedback has brought to light unique, crucial, and often silent perspectives from non-traditional sources. This feedback model has had a profound impact on resident satisfaction with rotations and significantly improved our overall feedback process with several meaningful changes implemented thus far. CONCLUSIONS REFERENCES 1.Andolsek KM, Nagler A, Dodd L, Weinerth JL. Internal reviews benefit programs of the review team members and the program under review. J Grad Med Educ 2010 Dec;2(4):604-9. 2.Arrighi JA. New Program Director Workshop: The Program Evaluation Committee and the Annual Program Evaluation. Accreditation Council for Graduate Medical Education (ACGME), 2015.