THE OSTE AS A FACULTY DEVELOPMENT TOOL 38th STFM Annual Spring Meeting, May 4, 2005 New Orleans, Louisiana University of California, Irvine School of Medicine Désirée Lie, MD, MSED, Penny Murata, MD, John Boker, PhD, Emily Dow, MD, Elizabeth Morrison, MD, MSED

OSTE AND FACULTY DEVELOPMENT Session Objectives By the end of the session, participants will be able to: Describe the OSTE and its uses. Discuss strategies for faculty development of community preceptors. Examine the role of the OSTE in community faculty development. Identify ways in which the OSTE may be useful in their own programs.

OSTE AND FACULTY DEVELOPMENT Session schedule Intro and background (10 mins.) Sample OSTE stations for comparison and discussion (20 mins.) Results of UCI OSTE (5 mins.) Audience feedback and suggestions (10 mins.)

USES OF OSTE Reported uses have been formative rather than summative Validated as assessment for residents only References FM clerkship faculty (Prislin et al. Acad. Med. 1998, 73:1116-18) Residents-as-teachers (Morrison et al Oct 2002, Acad. Med. 77:10;S29-32)

COMMUNITY FACULTY DEVELOPMENT Purposes: Orient preceptors to goals and students Increase networking between faculty and community teachers Improve teaching skills Retain and recruit Other purposes?

COMMUNITY FACULTY DEVELOPMENT Strategies (discuss pros and cons of each) CME-based teaching skills workshops Appreciation dinners with keynote speakers One-on one teaching consult at site visits Mailed materials (CD-ROMs, syllabi) Web-based programs

FACULTY DEVELOPMENT: OUTCOME MEASURES How do we measure program ‘success’? Student evaluations? Preceptor self-ratings? Retention and recruitment rate? Attendance? How often is teacher evaluation needed? When there is a problem/complaint? Regular interval assessment?

UCI COMMUNITY FACULTY PROGRAM ECE for MS2 FM, IM and Peds community faculty Heterogenous history of teaching Multiple roles and learners Need to retain and recruit Limited rewards

UCI COMMUNITY FACULTY PROGRAM Strategies (multiple) Noon series on teaching skills (6 hours CME) Evening and weekend EBM workshops (3 hours CME) OSTE workshop (3 hours CME) Site visits (teaching consults)

UCI OSTE Formative, used as teaching tool Adapted cases developed for residents using validated checklists CME-based (3 hours) Participants were also observers Cross-checked performance with observed real clinical teaching

UCI OSTE: CASES Based on most frequent teaching encounters and baseline needs assessment of faculty: Orienting the learner Bedside teaching Giving feedback Outpatient precepting EBM station Other possible stations: teaching charting, the difficult learner, teaching a procedure, inpatient teaching

UCI OSTE Show video of two teachers Bedside (PE) teaching case (Case C) Show video of two teachers Compare rating of teaching skills using checklist (audience) Match to other observations of teaching

UCI OSTE RESULTS Two OSTEs offered to 17 Preceptors 2004-5 Evaluations of OSTE itself Performance of preceptors Correlation with other measures of teaching Challenges Cost effectiveness

Evaluation by attendee preceptors (N=17) UCI OSTE RESULTS Evaluation by attendee preceptors (N=17) Question (Excellent, very good, good, fair and poor) Excellent to Very Good responses Overall evaluation of the activity 88% Quality of the educational content 100% Did this activity meet your need to upgrade your teaching skills? Was this activity relevant to your teaching? Did this activity increase your knowledge and/or skills in teaching patient care? Were the objectives met? (out of 3) 88-100%

UCI OSTE RESULTS Comments from preceptor participants: Question: What did you like most about the format of the program? Ability to take different roles (Teacher vs Observer) Workshop was primarily interacting with students (very hands on) Question: How will you change your teaching practices as a result of this activity? Be more observant and don’t assume what the student know Ask more questions and have student take more responsibility for learning Listen to medical student more and offer more positive feedback

UCI OSTE RESULTS Linking Preceptors and Students through Education Summary of Case Encounters Inter-Rater Agreement (IRA) of Ratings of Preceptor Case Performance1 Case A: Orienting the Student Preceptor ID# # Rater IRA 89 3 .337 48 .447 87 2 .614 74 .781 57 .779 70 (-)2 Mean .592 Case B: Outpatient Precepting .504 Case C: Bedside Teaching .439 Case D: SOAP Note Feedback .403 Mean IRA across Cases and Preceptors3 .485 1Computed as intraclass correlation coefficient (ICC), using the consistency definition and a two-way random effects model where both measures effects and raters effects are random (SPSS 12.0 Reliability Analysis). 2Negative value resulting from negative average covariance among items. 3Computed only for obtained ICC values that were positive.

Inter-rater agreement on station UCI OSTE RESULTS Inter-rater agreement on station

Correlation with other measures UCI OSTE RESULTS Correlation with other measures Student evaluations of preceptors Preceptor self-ratings of teaching skills Site visit assessments of actual teaching by UCI faculty (DL, ED and PM)

UCI OSTE RESULTS: CORRELATIONS ID Self-Rating (Global) OSTE Score (Mean Item Rating) Global Teaching Score (Site Visit) 89 4 3.44 106 3 3.81 5 73 - 3.64 2 48 4.22 87 3.90 74 3.87 57 4.81 70 4.33 83 3.37 36 3.79 29 3.78 62 4.58 102 3.39 71 3.10 67 4.11 N = 15 13 15 9

Motivation (CME? Networking? Fun?) Accuracy of observed performance: UCI OSTE: CHALLENGES Attendance: Timing, punctuality Motivation (CME? Networking? Fun?) Accuracy of observed performance: Participants also observe stations using checklists Stations not ‘blinded’ Optimal number of stations for performance vs practice?

UCI OSTE: COST EFFECTIVENESS Costs include: Use of training center and staff Fee for ‘standardized students’ ($75 each) CME fees, food, parking $$ spent: $1500 for OSTE (10 to 16 preceptors) $400 for food (attendees, students, faculty, staff) Faculty time

UCI OSTE: UNANTICIPATED OUTCOMES No shortage of SSs Students enjoyed the event enormously Preceptors pleasantly surprised by lack of didactics New preceptors asked for preparation Networking between students and preceptors

OSTE AND FAC. DEV: AUDIENCE FEEDBACK Questions: Would you use OSTE? If so, in what setting? How would you modify for your setting? Comments?

OSTE AND FAC. DEV: LESSONS LEARNT PERFORM NEEDS ASSESSMENT AGGRESSIVE MARKETING LOW STAKES FOR PRECEPTORS (FORMATIVE) INCLUDE UNIVERSITY FACULTY (MIX) NEED OTHER OUTCOME MEASURES MAKE IT FUN!

OSTE AND FACULTY DEVELOPMENT For more information: Désirée Lie, MD, MSEd, Family Medicine Email dalie@uci.edu Penny Murata, MD, Pediatrics Email: prmurata@uci.edu Supported by a primary care faculty development HRSA grant from the Bureau of Health Professions, #D55HP03362-01-00, 2004-7