Conducting Your Program’s Annual Review Esther Tucci, CTAGME Pinnacle Health Harrisburg, PA

About Pinnacle Health GME Internal Medicine Residency – 35 General Surgery Residency MD – 13 DO – 6 Toxicology Fellowship - 1 Bariactric Surgery Fellowship – 1 Osteopathic Programs FP – 3 IM – 3 Orthopedics – 12 – Intern - 12

What is an Annual Review? RRC of Surgery Program Requirements VI.C – Program Evaluation “The educational effectiveness of a program must be evaluated at least annually in a systematic manner”

Annual Review vs. Internal Review Annual Program/Common Requirements Done Annually Conducted by program Program representatives Plan of action approved by faculty and documented in minutes Internal Institutional Requirements Done midpoint between ACGME accreditation surveys Conducted by GMEC External representatives A written report by the GMEC and required to give to ACGME site visitor for Institutional Reviews

Definition of Annual Review Internal program personnel, i.e. PD, Faculty and at least one resident Review program goals and objectives and the effectiveness of the program Must have documentation of the meeting at least annually Must evaluate written comments from the faculty, most recent GMEC report, residents confidential written evaluations

Definition cont’d…. Use resident performance and outcome assessment in its evaluation of the educational effectiveness of the residency program Program should have a process in place to use these assessment results to improve the residency program If deficiencies are found the group should prepare a plan of action approved by faculty and documented in the minutes

WHAT WE DID………..

Formed Our “group” PD All paid faculty Two residents Invite all volunteer faculty – those that responded attended Administrative/DIO (VP of Medical Affairs)

Location and Time Did survey on preference of date and time Off campus ½ day time commitment – (breakfast/lunch) 730 am to 1200 noon Close to the hospital

Evaluation Sources Used in the Review Faculty Eval of Prog Resident Eval of Prog Resident Eval of Faculty Resident Eval of Services ACGME Resident Survey Summary Duty Hour Reports Medical Student Clerkship Evaluations M&M Conference Evals Last GMEC Internal Review Last RRC Site Visit Results Last Institutional Review

Evaluation Sources Used in the Review Adequacy of clinical experience: 5-year summary of ABS Exam Results Resident ABSITE Score Results Graduate Surveys Resident SOL data Resident bedside procedures

Evaluation Sources Used in the Review ACGME Competency Assessment Surgery Competency Charts

Pre-meeting Work Assignments Each member of the “group” received the following pre-meeting work assignments: Copies of all evaluation sources for their review S.W.O.T Form Faculty Development Survey Items of Discussion List

S.W.O.T Form Please list what you feel are the residency program’s…. “S”trengths “W”eaknesses “O”pportunities “T”hreats

Needs Assessment for Faculty Development Each member was asked to list their interest in learning more about the following topics: Teaching Evaluation Technology Research Career Development Rate his/her preference for session times

Items of Discussion List Need to develop this list to fit your own program needs Examples of some items for Pinnacle: Incorporating competencies into goals/objectives Improving standards for our Selection Policy and criteria Enforcing higher standards for Promotion and Resident Evaluation Policy Recruitment of FT Faculty On-line curriculum for ABSITE review New Academic Curriculum 3-chief year July 06 – June 07

Working Binder Each member of the group received a binder ahead of time that included the following: Pre-work assignments Evaluation sources Copies of Program Requirements Current Rotation Goals & Objectives Program Policies/GMEC Policies Definition of Core Faculty Each member was instructed to bring this binder with them to the meeting

At the Meeting Agenda S.W.O.T form results Review items off Discussion List Documentation and Action Plan Based on the deficiencies or outcomes from first two items New Initiatives

Meeting Documentation DISCUSSION And RRC REQUIREMENT (**) ACTION PLAN/GOAL Full Time Faculty numbers Faculty manpower needs, funding for faculty ** In addition to the PD for each approved chief residency position, there is a least one additional geographic full time teaching staff member whose major function is to support the residency program. Pursue Critical Care Surgeon for full time faculty. Will no longer pursue a Trauma Surgeon. Need to know from Administration the funds available for recruiting and salary. Find out what the program’s reimbursement amount is and how the money is used. Need support from Administration for aggressive recruitment to meet this need.

Meeting Documentation DISCUSSION And RRC REQUIREMENT (**) ACTION PLAN/GOAL Lack of Trauma Protocol for HH Lack of trauma alerts coming to HH ER Some senior and all junior residents will not meet their numbers in trauma due to the change of no trauma alerts coming to HH ER ** Residents are required to have 10 operative trauma cases and 20 non-operative trauma cases. Will begin to send residents to Shock Trauma for one-month at a PGY 2 level and one month at a PGY 4 level. May need to add an additional month For more trauma and/or surgical critical care

Approval of Plan of Action We then submitted our “Plan of Action” at the next faculty meeting for discussion and approval which was documented in the minutes A summary of the Annual Review outcome was then forward to the GMEC for review and to address any concerns or issues.

Timeline You want to do this early enough so that changes or deficiencies can be addressed for the upcoming academic year starting in July

Follow Up We planned a follow up meeting 3-6 months after the meeting to give updates or failed attempts to our action plan Will continue to hold an annual review meeting each year ~ March/April

Tips or Variations for your Own Annual Review First, you must know what your RRC requires in regard to your review of the program Set up a template to meet your own program needs and/or deficiencies and if you have no deficiencies noted then use it to plan new initiatives

Tips or Variations for your Own Annual Review Use the template as your meeting agenda to make sure all areas are discussed and use it to write your minutes for documentation Distribute the minutes to all faculty and residents even those not in attendance at the meeting

Tips or Variations for your Own Annual Review Keep a copy of the meeting minutes handy for your next RRC accreditation site visit Make this a “living document” and plan follow-up meetings in 3-6 months Evaluate what has happened to any plans What have been the successes What have been the challenges

Tips or Variations for your Own Annual Review Most of us already have multiple opportunities to review our residency programs…Faculty Meetings, Internal Curriculum Committee Meeting, schedule Retreat/Planning Meetings. Once you identify the strategy that works best for your program, remember to thoroughly document the review.

Questions

Special Thanks to: Michael Holman, MD Raymond Kostin, MD