The Educator’s Portfolio: Creation and Evaluation David M. Irby, PhD Kanade Shinkai, MD PhD Patricia O’Sullivan, EdD University of California, San Francisco Academy of Medical Educators

Preview Identify the major roles of educators Determine documentation standards and evaluation criteria for each role Examine the Academy of Medical Educator’s Portfolio and begin creating your own portfolio Review UCSF promotion processes and how a portfolio relates to Advance

Agenda for the session Introduction/Preview – educator roles (20 min) Presentation on evaluation constructs (30 min) Review and evaluation of a portfolio Small group work—Exercise 1, 3 Parts (25 min) Report Out (10 min) Small group work—Exercise 2 (20 min) Break (15 minutes) Creating your UCSF portfolio (45 min) Presentation - Advancement and Promotion (60 minutes)

Question What are the core roles of an educator? Think, pair, share Slides available at https://wiki.library.ucsf.edu/display/UCPLS/The+Educator%27s+Portfolio Educator Roles Evaluation Criteria Portfolio Review Advancement

Curriculum Development Educator Roles Teaching Activities Curriculum Development Learner Assessment Mentoring & Advising Educ Leadership & Administration Educator Roles Evaluation Criteria Portfolio Review Advancement

Rationale for Educator’s Portfolio Documents achievements, reflections and plans Makes work visible and therefore peer reviewable Enables education to be scholarly Public Peer reviewed Platform to build upon Educator Roles Evaluation Criteria Portfolio Review Advancement

Timeline of National Initiatives Education as Scholarship - 1990 Boyer - 1997 Glassick - 2000 Fincher Documentation standards for Educational Scholarship - 2007 Simpson Criteria for Evaluating Educational Scholarship - 2013 Gusic Educator Roles Evaluation Criteria Portfolio Review Advancement

The Scholarship of Teaching Scholarship includes Discovery Integration Application Dissemination/teaching Educator Roles Evaluation Criteria Portfolio Review Advancement

Glassick’s Criteria for Scholarship Clear goals Adequate preparation (e.g., work is informed by the literature and best practices in the field) Appropriate methods Significant results Effective presentation Reflective critique Educator Roles Evaluation Criteria Portfolio Review Advancement

Documentation standards for Educational Scholarship 5 domains of educator activity Documentation requires Quantity Quality Scholarly teaching (builds on others) Scholarship of teaching and learning (advances) Simpson D, et al. Advancing Educators and Education: Defining the Components and Evidence of Educational Scholarship. 2006 Available at: aamc.org Educator Roles Evaluation Criteria Portfolio Review Advancement

Excellence and Scholarship Criteria Teaching Excellence Scholarly Teaching Scholarship of Teaching Quantity & Quality Builds on Field Public Peer Review Contributes to Field Simpson D, et al. Advancing educators and education by defining the components and evidence associated with educational scholarship. Med Educ. 41:1002–1009, 2007. Educator Roles Evaluation Criteria Portfolio Review Advancement

Kirkpatrick’s Model to Judge Results Kirkpatrick Level Assessment Level Criteria in toolbox 1 Satisfaction/reaction Of participants/stakeholders 2 Learning Measures of knowledge, skills, attitudes, and/or behaviors 3 Application Desired performance demonstrated in other settings 4 Impact On education programs and processes within and/or outside institution Kirkpatrick DL and Kirkpatrick JD. Evaluating Training Programs: The Four Levels (3rd Ed); 2006. Educator Roles Evaluation Criteria Portfolio Review Advancement

AAMC Toolbox for Evaluating Educators Glossary Executive Summary User’s Guide Toolbox with Tables of Indicators for Evaluation Appendices Reference list Gusic M, Amiel J, Baldwin C, Chandran L, Fincher R, Mavis B, et al. Using the AAMC Toolbox for Evaluating Educators: You be the Judge!. MedEdPORTAL; 2013. Available from: www.mededportal.org/publication/9313. Educator Roles Evaluation Criteria Portfolio Review Advancement

Small group work: Exercise 1 Part A Review the teaching activities section of the faculty member’s portfolio Discussion How would this colleague be evaluated at UCSF? Report out Key points from the discussion Educator Roles Evaluation Criteria Portfolio Review Advancement

Exercise 1-Continued Part B Discussion Report out Use the criteria from the toolbox to evaluate the teaching activities in the portfolio Discussion How do the criteria help you show that this faculty member’s work should be valued? Report out Which elements of the tool were most helpful to you? Educator Roles Evaluation Criteria Portfolio Review Advancement

Exercise 1-Continued Part C Review the narrative summary about the educator that was created using the criteria from the toolbox Large group discussion—Comments/impressions Educator Roles Evaluation Criteria Portfolio Review Advancement

Exercise 2 Review curriculum development portfolio Discussion Use the criteria from the toolbox to evaluate the portfolio entry Discussion Identify areas of strength and targets for improvement Report out Which elements of the tool were most helpful to you?

Curriculum Development Educator Roles Teaching Activities Curriculum Development Learner Assessment Mentoring & Advising Educ Leadership & Administration Educator Roles Evaluation Criteria Portfolio Review Advancement

Break

Elements of a UCSF Portfolio Personal statement Documentation in one or more areas Direct Teaching Program Design & Curricular Innovation Advising & Mentoring Educational Administration & Leadership Educational Research http://medschool2.ucsf.edu/academy/Educators_Portfolio/ Educator Roles Evaluation Criteria Portfolio Review Advancement

Direct Teaching

Preparing your Educator’s Portfolio The Educator's Portfolio Faculty Advancement - Navigating UCSF Preparing your Educator’s Portfolio Choose categories of educational activity Direct teaching required for AME application plus one other area Start collecting data early Write your personal statement Request a trained AME coach This really needs a lot of work – please help Educator Roles Evaluation Criteria Portfolio Review Advancement Jessica Muller, PhD Martin Bogetz, MD 22

Personal Statement for UCSF Academy Addresses the applicant's educational philosophy, professional development in education, and intended contributions to the Academy and to the School of Medicine (2 pages maximum) Write a personal statement Educator Roles Evaluation Criteria Portfolio Review Advancement

Resources These Slides UCSF Academy of Medical Educators Portfolio https://wiki.library.ucsf.edu/display/UCPLS/The+Educator%27s+Portfolio UCSF Academy of Medical Educators Portfolio http://medschool2.ucsf.edu/academy/Educators_Portfolio/ AAMC Toolbox Gusic M, Amiel J, Baldwin C, Chandran L, Fincher R, Mavis B, et al. Using the AAMC Toolbox for Evaluating Educators: You be the Judge!. MedEdPORTAL; 2013. Available from: www.mededportal.org/publication/9313.

Portfolio and Advancement Educator Roles Evaluation Criteria Portfolio Review Advancement