1. 1. Meaningful rubric could facilitate honest and useful evaluations Consistent standard- Mirroring ACGME Milestones approach. Emphasize constructive, specific comments. 2. Training on evaluations process may confer reliability among evaluators Remove concerns re: anonymity and retaliation. Focus on purpose of evaluation. Reinforce need for constructive criticism. 3. Promotion and Tenure Process Transparency Consider track and qualification of learners. Consider the competing priorities faculty face depending on their academic track and departmental mission. Theory & Conclusion How do clinical teaching faculty interpret feedback from rotators? Tara F Carr MD, Guadalupe F Martinez, PhD AMES-OMSE Teaching Scholars Department of Medicine, University of Arizona College of Medicine, Tucson AZ Introduction Recommendations Saturated Themes (cont) References 1. Epstein RM, Siegel DJ, Silberman J. Self-monitoring in clinical practice: a challenge for medical educators. J Contin Educ Health Prof. 2008 28(1):5-13. 2. Arah OA, Hoekstra JAB, Bos AP, Lombarts KM. New tools for systematic evaluation of teaching qualities of medical faculty: results of an ongoing multi-center survey. PLoS ONE 2011, ;6(10):e25983. 3. Overeem K, Driessen EW, Arah OA, Lombarts KM, Wollersheim HC, Grol RP. Peer mentoring in doctor performance assessment: strategies, obstacles and benefits. Medical Education 2010, Feb;44(2):140-7. 4. Corbin J, Strauss A. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. SAGE Publications, Inc, 2007. 5. Creswell JW, Miller DL. Determining Validity in Qualitative Inquiry, Theory Into Practice, 2000; 39:3, 124-130. Self-monitoring is a challenge for medical educators. 1 Providing concrete plans and structured assessment is more helpful for faculty improvement than independent change. 2,3 While giving and receiving feedback is emphasized through Graduate Medical Education, there are barriers to the use of that feedback on the part of the faculty. Purpose of the study: Describe and form a theory about how Medicine faculty interpret written evaluations from clinical rotators. Describe and form a theory about whether and how Medicine faculty use feedback to inform their own educational practice and philosophy.. Overall Research Questions Overall Research Questions 1. Method of Feedback is Problematic Current grading scale is poor- 7/12 Desire for more substantive feedback- 6/12 2. Concerns with Evaluators Poor feedback from evaluators- 8/12 Not enough time spent with the evaluator Evaluators may be scared to give honest feedback Evaluators don’t care about evaluations process Evaluators need education on the process and identifying teaching moments 3. Perceptions Regarding Evaluation use in Promotion & Tenure Process Reviews Conducted within the Context of Clinical Responsibilities and Academic Contract is Important- 5 (Assist Prof) Filtering of Evaluations Through Relevant Superiors or Averaged Over Time is Highly Valued- 6 (1 Prof; 3 Assoc Prof; 2 Assist Prof) Methods Qualitative Methodology - Grounded theory 4 Purposeful selection Triangulation for Qualitative reliability and validity 5 1. 12, 15-60 minute one-on-one semi-structured open ended Interviews 2. Observation reflection notes Analysis with Nvivo10© Coding/Data Reduction → Research Site and Sample 12 Subspecialty faculty in a division within the Department of Medicine at a southwestern academic center. Faculty appear to interpret evaluations based on perceptions they have about feedback methods and the evaluator’s ability to assess and recognize teaching skills. Moderate use in informing practice. The perceived current role of evaluations in P&T process further shapes the educational philosophy of these faculty members. Limitations - Small sample size; interviews not transcribed verbatim; limited to Medicine faculty perspective; overrepresentation of junior faculty. Saturated Themes 3-Selective →Inform practice 2-Axial →Issues/Reaction 1-Open →Broader perceptions Years experience as faculty member at site: Range: 0.5-24 Average 6.58 Do clinical teaching faculty use learner evaluations to inform their teaching practices? What are their interpretations of the value of these evaluations, and what are implications on their own development? “The verbal feedback from my colleague/boss made me more cognizant of my behavior and I modified it appropriately.” “I think if you are an academic physician with a teaching role, the evaluation should … [be used, but] taken in context. People who are mostly clinical should be held to a higher standard…” “Inexperienced rotators might not know when the teaching is happening.” “I think the comments a the bottom in free text, under the numbers are helpful.” The numbers are worthless. If someone gives me a 7 or 8 with a baseline to get all 9s; if I get a 7, am I really bad at something or is 7 still technically superior?”