Meeting the FM RRC Faculty Development Requirements: Curriculum and Program Evaluation Highlights from Year 1 44th Annual STFM Spring Conference April 29, 2011 Joseph J. Brocato, PhD, Mark Yeazel, MD, MPH & Erik Solberg, MA

Objectives By the end of the session, conference attendees will be able to: Explain the elements (steps) for a systematic approach toward developing and evaluating a longitudinal residency-based faculty development curriculum List several examples of practical methods toward meeting continuing faculty development requirements, e.g. overall program design, instructional content, program evaluation Describe the content of an electronic tool useful for conducting a local needs assessment of faculty Identify and share national, regional, and local faculty development resources that can “jump-start” the development of a local faculty development within residencies

The Importance of Faculty Development for Family Medicine The constant need to train new faculty in family medicine, sometimes directly out of residency Midcareer faculty may have had uneven access to title VII faculty development training Senior faculty often express an interest in returning to teaching and academic toward career capstones Accreditation: ACGME RRC Requirements….

The ACGME Requirement for Faculty Development “There must be a structured program of faculty development that involves regularly scheduled faculty development activities. Since family medicine faculty should demonstrate the same skills, knowledge and attitudes that are expected of the residents, faculty skill development and update are an important part of faculty development. The program is expected to address clinical, educational, administrative, leadership, research and behavioral components of faculty performance.” FM RRC Requirement II.B.9

The ACGME Requirements for Faculty Development (continued) “It should involve at least annual departmental, residency and individual faculty needs assessments, and may include structured group and individual activities. Although clinical update is important, faculty development should provide experience to improve teaching in all settings. This should be measurable and documented in evaluations by residents”.

The Department of Family Medicine & Community Health at The University of Minnesota Founded in 1971 with over 1,630 residency alumni Department: Located on TC campus, 73 fulltime faculty and 994 non salaried community faculty Eight Residencies: five TC urban & suburban sites, plus three rural and small town sites--Duluth, St. Cloud, Mankato 166+ Residents Two fellowships: Sports Medicine and Hospice and Palliative Medicine

Kern Curriculum Development Model Problem identification done for us! Targeted Needs Assessment first phase Goals and Objectives for the Fellowship Followed

The curriculum: Determining the Target Audience Target audience: new/junior faculty; 19/77 of faculty had 0-5 years or less of teaching experience. This cohort is the largest 5 year block and has increased since needs assessment conducted Prior fellowship and training for mid- career and senior faculty existed Open to all faculty MD, Ph.D. and D.O faculty—all have needs for development Open to UME and GME faculty

The curriculum: The Second Year Class 32 total graduates in first two years 13 faculty fellows, 7 in system, 6 out of system. One is fellowship-based, 12 are residency-based One DO, one PharmD, and 11 MDs. 60% Female and 40% Male Diversity represented: URM 30%

The curriculum: Fellowship’s Goals The faculty development fellowship program’s goals are to teach faculty development fellows how to: Construct needs assessments, goals and objectives, educational and evaluation strategies, and evaluation plans (i.e., systematic instructional development) toward the implementation of local curricular content. Integrate the principles of educational theory toward shaping personal educational theories and educational best practices implementation. Develop teaching skills mastery (lectures, small groups, etc.) with structured coaching and peer feedback. Incorporate instructional technology where appropriate to address instructional delivery limitations and challenges.

The curriculum: Selected Fellow’s Curricula Projects Acquisition of Pain Management Skills Coding & Billing & QI* Breastfeeding Skills Antibiotic Selection* Guiding Patient Center Behavior Change: Motivational Interviewing* Transformative Journal Club/EBM Hospital Medicine Track* Outpatient Procedures Musculoskeletal Manual Med. For DO's Team-Based Care Model Colposcopy Course - Outpatient Gynecology –*collaborative curriculum

The curriculum: Instructional Content Six monthly 3 hour workshops (18 hours total), Tuesdays 2-5pm Two to three educational topics presented monthly; mini-lecture format Structured small group curriculum development time Required text: American College of Physicians Teaching Medicine series (6 volume set, 2010 copyright).

The curriculum: Instructional Content (Continued) Six Monthly Workshops Taught by Joseph Brocato Ph.D., Mark Yeazel MD MPH, & Erik Solberg MA –January 2011: Curriculum Development Overview; Targeted Needs Assessment –February 2011: Basic Educational Theory; Writing Goals and Objectives, Career Development –March 2011: Teaching Strategies; Office-Based Teaching Strategies; Formal Presentation Skills –April 2011: Learner Evaluation; Mentoring Skills; Teaching with Demonstrations –May 2011: Program Evaluation; Small Group Teaching; Leadership Skills –June 2011: Fellowship Curriculum Project Presentations

The curriculum: Instructional Methods Workshops with formal presentation, small group curriculum development, small group discussion of teaching Moodle website for background readings, curriculum examples, essential course documents, formal presentation archives

Evaluation A retrospective pre-post self-assessment of Academic Skills –40 items in curriculum development, learning theory, evaluation, technology, teaching strategies, formal presentation skills, small group discussion, clinical teaching encounter = our objectives.

Evaluation Post course scores range from 3.5 to 4.32 on 5 point scale* Highest scored items: (1) Write course goals; (2) Demonstrating effective use of lecture; (3) Describe rationale for objectives Lowest scored items: (1) Predict effect of principles of learning in different situations; (2) Apply principles of learning and motivation; (3) Explain principles of learning and motivation –*5=extremely able, 4=Highly Able, 3=Able, 2=Slightly Able, 1=Not at all able

Evaluation Significant improvement was seen for all objectives ObjectivePre-post difference Sig.(2-tailed) List steps in systematic approach to instruction Write educational objectives Design evaluation Explain principles of learning Demonstrate use of questions in lectures Demonstrate use of feedback in small groups

Evaluation Pre/Post Teaching Formative Evaluation –Lecture Presentation Skills within Medical School, Residency, Fellowship –Preceptor encounters in clinic Post workshop session 1- minute evaluations

Evaluation Data Trends: Years 1 and 2 Lessons Learned Session evaluations show workshop formats are optimal, but like to vary order of elements. Curriculum development is a “new language.” Some are frustrated that topics are presented as overviews Scheduling two observations for 20 faculty impossible given geographic distance, schedules, teaching faculty time available

Evaluation Data Years 1 and 2: Lessons Learned Faculty appreciate joint projects and fellowship flexibility (curriculum implementation not expected during 6 month fellowship) Writing curriculum in workshops in sporadic Fellows enjoy discussing their teaching and sharing ideas and resources in addition to curriculum development

Faculty Development Needs Assessment Survey created using Survey Monkey Conducted online 77 respondents of 98 faculty = 79% response rate

Needs Assessment Survey Items in four domains –Education –Research and Written communication –Administration –Professional Academic Skills Items ranked on 3 point scale –1 Little need/interest for faculty development –2 Moderate need/interest –3 High interest/need

Needs Assessment: Sample Questions

Needs Assessment: Education Selected Items Evaluating teaching effectiveness Evaluating learner performance Using the Web for course development Clinical teaching (e.g. precepting or bedside) Giving learners positive and negative feedback

Needs Assessment: Research and Written Communications: Selected Items EBM and critical appraisal of literature Writing peer-review journal articles Writing grant proposals Grant opportunities for academic family physicians

Needs Assessment: Selected Items Leadership skills for academic family physicians Overview of the UMN medical School, and Academic Health Center Promotion and tenure ACGME and FM RRC: Program requirements, PIFS, Internal Reviews Residency management systems: E-Value and New Innovations

Needs Assessment: Professional Academic Skills: Selected items How to advise medical students and residents How to be a mentor Career development for family physicians Finding and receiving faculty mentoring Establish a healthy work life balance

Needs Assessment Number of years you have taught? What types of learners do you teach? … percentage of time I spend teaching in the follow settings (lecture, small group, precepting, etc.)? Preferred delivery methods, times? Participated in other faculty development activities?

RankSubject AreaLittle NeedModerate need High needRating Average 1Leadership skills for academic family physicians (A) 24.3% (18)33.8% (25)36.5% (27)2.13 2Evaluating teaching effectiveness (E) 21.6% (16)43.2% (32)33.8% (25)2.12 3Evaluating learner performance (E)23.0% (17)41.9% (31)33.8% (25)2.11 4How to be a mentor (P)20.3% (15)48.6% (36)29.7% (22)2.10 5Evaluating program effectiveness (E) 21.6% (16)51.4% (38)29.7% (22)2.08 6Establish a healthy work-life balance (P) 28.4% (21)36.5% (27)35.1% (26)2.07 7Dealing with the problem medical students/residents (E) 24.3% (18)43.2% (32)29.7% (22)2.06 8Motivating medical students/residents to learn (E) 23.0% (17)47.3% (35)27.0% (20) (T)How to advise medical students and residents (P) 31.1% (23)36.5% (27)28.4% (21) (T)Using the web for course development (e.g. Moodle) (E) 32.4% (24)35.1% (26)29.7% (22)1.97

RankSubject AreaLittle NeedModerate need High need Average 11 (T)Grant opportunities for academic family physicians (R) 33.8% (25)31.1% (23)29.7% (22) (T)Writing grant proposals (R)35.1% (26)31.1% (23) (T)Giving learners positive and negative feedback (E) 29.7% (22)43.2% (32)25.7% (19) Career development for family physicians (P) 27.0% (20)45.9% (34)21.6% (16) (T)Electronic resources for academic family physicians (R) 29.7% (22)44.6% (33)23.0% (17) (T)Writing peer-reviewed journal articles (R) 35.1% (26)33.8% (25)28.4% (21) Evidence based medicine and critical appraisal of the literature (R) 32.4% (24)41.9% (31)24.3% (18) (T)Clinical teaching (e.g. precepting or bedside) (E) 33.8% (25)37.8% (28)24.3% (18) (T)Developing instructional materials (E) 28.4% (21)51.4% (38)18.9% (14) (T)Finding and receiving faculty mentoring (P) 32.4% (23)44.6% (33)21.6% (16) (T)Teaching and evaluating the ACGME General Core Competencies (E) 28.4% (21)50% (37)17.6% (13)1.89

Constraints Toward Local Faculty Development $$ for program operations Lack of educational expertise: in development and in teaching Competing faculty demands, primarily clinical/Encroachment on administrative time for faculty Lack of administrative time to implement Other Piece-meal faculty development locally, regionally, nationally Others?

Getting Buy-In Supportive department chair and residency program directors Supplementing local, regional, national faculty development Agreement to repeat based upon interest Close relationship among area residency programs outside the system

Putting a price on faculty development Total Year one Budget (20 fellows): $14,686 –Faculty time ($9,246) –Textbooks ($1,500) –Handouts/Copying ($1,000) –Snacks ($1,200) –Parking ($1440) –Graduation Certificates ($200) –Misc. Supplies ($100)

Putting a price on faculty development (continued) The power of collaboration: cost- sharing for outside of the system faculty ($14,685/20=$ per trainee) –$14,685/20=$ –Less than $100 per instruction hour—compares favorably with CME and university tuition

Constraints and solutions? How are you meeting the Faculty Development RRC requirements? Constraints that prevent you from meeting the RRC requirements? Solutions? –National –Regional –Local

Conclusions/Next Steps for Our Program Complete second fellowship year and prepare for year three Continue to collect evaluation data Consider longitudinal evaluation items for year three Mobilizing fellowship alumni