Implementation and Development of Learner Portfolios Stacy Potts, MD Annual Spring STFM Conference New Orleans April 30, 2011

Objectives Conference participants will: Examine the need for a learning portfolio in graduate medical education Determine the general elements of a learning portfolio Understand the types of learner portfolios used by Family Medicine residencies nationally

Blueprint - Methods Why portfolios? What are portfolios? How are portfolios best? –Successes and challenges

The Forest

Our mission To train competent family physicians to improve the health of patients

Changes in GME From Process to Outcomes From Curriculum Guide to Evaluation Data From Program Evaluation to Individual Resident Outcomes Not enough to teach it – it must be learned AND NOW… From documenting experiences to proving competence

ACGME Outcomes Project Home Welcome The Outcome Project Our Goals Medical Knowledge Patient Care Practice-based Learning Systems-based Learning Professionalism Interpersonal andCommunication Skills ACGME Expectations Program Requirements Timelines Summary

Accreditation ACGME Competencies – The Outcome Project –Patient Care –Medical Knowledge –Practice-based Learning and Improvement –Interpersonal and Communication Skills –Professionalism –Systems-based Practice

Competency Based Assessment Requires demonstration of application of a body of knowledge rather than knowing about a body of knowledge A competency is a statement of learning outcomes for a skill or body of knowledge. –students demonstrate their ability to do something –show the outcome of the learning process.

Competence “The habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individuals and communities being served.”

Integrated Competency Diagram (H. Bell) MedicalKnowledge Patient Care Professionalism Interpersonal and Communication Skills Practice-based Learning System-basedPractice

Integrated Competency Diagram MedicalKnowledge Patient Care Professionalism InterpersonalandCommunicationSkills System-based Practice Practice-based Learning and Improvement

"The hardest conviction to get into the mind of a beginner is that the education upon which he is engaged is not a college course, not a medical course, but a life course, for which the work of a few years under teachers is but a preparation." - William Osler

IOM Patient Centered Care Evidence Based Medicine Reflective Practice

Our Mission “To train competent family physicians in a supportive learner-centered environment to care for communities with a patient centered, evidence based approach focusing on the development of skills in lifelong learning and reflective practice.”

Experiential Learning Concrete Experience Reflective Observation Abstract Conceptualization Active Experimentation Kolb Learning Cycle Observed Behaviors What ? So what? Now what? Private Learnin g Process Kolb D.A. (1984) 'Experiential Learning experience as a source of learning and development', New Jersey: Prentice Hall

Our Setting UMass Worcester Family Medicine Residency Program –3 year training program to attain eligibility for board certification in the specialty of family medicine –12 residents per year divided between three health center sites

Integrated Competency Diagram MedicalKnowledge Patient Care Professionalism InterpersonalandCommunicationSkills System-based Practice Practice-based Learning and Improvement

Practice Based Learning and Improvement Residents must demonstrate the ability to investigate and evaluate their care of patients Continuously improve based on constant self evaluation and lifelong learning –Set learning and improvement goals –Identify and perform appropriate learning activities –Incorporate formative evaluation feedback –Use IT to optimize learning

Portfolios: A Definition A collection of materials, especially when representative of a person’s work: a photographer's portfolio; an artist's portfolio of drawings. »Dictionary.com

Learning Portfolio Structured collection of evidence that demonstrates progressive learning –“the perfect vehicle for the synthesis of adult learning and reflective practice.” »Maggie Challis –“a dynamic celebration of learning…” »Helen Barrett

Purposes and Uses of Portfolios Program Requirements Organizational or Professional Documentation Personal Development Social LEarning

Learning Portfolio Benefits Learner-centered Serves to ground, guide and assess learning, often through review with the faculty Provides evidence of application of learning Promotes increase in self-awareness, reflection, and self evaluation

Portfolio Spectrum Teacher Centered Teacher Driven Learner Centered Learner Driven Mixed Model Teacher and Learner Collaboration Little or no learner involvement Learner completely in control of design and content

Portfolio-based Learning Values past learning and experience Recognizes learners ability to learn and act independently Centrality of reflection to the learning process Allows meaningful links between experiences »Challis 1999

Outcomes Portfolios are not solely for assessment nor solely for documentation of competency Portfolios are a method of education Portfolios model continued professional development that is based in outcomes not merely in number of hours A portfolio’s purpose is to demonstrate learning, not to chronicle experiences

Portfolios in Medical Education Systematic literature review Factors of success –Clearly communicated goals and procedures –Integration with curriculum and assessment –Flexible structure –Support through mentoring –Measures to heighten feasibility and reduce required time Dreissen E et al. “Portfolios in medical education: why do they meet with mixed success? A systematic review.” Medical Education 2007; 41:

Portfolio Effectiveness Systematic literature review – articles from 10 countries involving seven healthcare professions Well implemented portfolios are effective and practical Electronic versions better at encouraging reflection Regular feedback from a mentor enhances the success Summative assessment best when triangulated with other assessment methods Tochel C et al. “The effectiveness of portfolios for post-graduate assessment and education. BEME Guide No 12” Medical Teacher 2009; 31:

Stages of Evidence Cycle Stage 1: Select your aspirations for good practice Stage 2: Set the standards for your outcomes – for being competent and providing a good service Stage 3: Identify your learning needs and identify your service needs then rank them in order of priority Stage 4: Make and carry out a learning and action plan with a timetable for you and for service development Stage 5: Document your learning, competence, and performance and standards of service delivery

Calibrating the Self Assessment Reviewing self assessment –Strengths –Opportunities for improvement Reviewing external evaluator’s assessment Building upon themes Reconciling incongruence

Collaborative Goal Setting Negotiate priorities Develop action steps Develop follow up plan FOLLOW UP

Portfolio Assessment Goals Provide feedback to learners, teachers, course directors Motivate learners and focus their sense of achievement Enable learners to remedy deficiencies Consolidate learning Estimate potential to progress to other levels Provide information to internal and external agencies

Assessment Builds on goals –Formative, summative Consider qualitative, quantitative Design new measures or use existing –Think outside the box Consider resources and time

Barriers Time –Learner, faculty (to do, manage, evaluate) Advisors –Availability, development Buy-in Lack of research Start-up resources Evaluation methods Technologic skill

Resident Portfolio Medical Knowledge Patient Care Interpersonal Skills and Communication Problem Based Learning and Improvement Systems Based Care Professionalism

Resident Portfolio Additional components Video taped encounters Reflective pieces Research development Community Service

ACGME Plans Competency–Based Portfolio Advisory Committee Charge: –Examine the need for a learning portfolio –Determine the general elements –Determine attributes of each element –Establish helpful principles for development –Recommend the types of policies for implementation

Committee findings The ACGME web based portfolio will: –Chronicle experiences and track progress against defined learning objectives –Support the development of self-directed habits of lifelong learning and reflective practice –Function as a repository for all structure evaluations of performance –Allow the resident to monitor his or her own learning and growth

ACGME Learner Portfolio Infrastructure for organizing the resident’s experiences and for gaining feedback on progress –Expectations –Evidence –Resident Reflections –Assessments (Formative and summative) –Integration with other ACGME systmes

Timeline 2007 – Alpha testing 2008 – Beta testing 2010 – Initial roll out 2012 – Voluntary broader use 2016 – Full implementation

Next steps

Conclusions Flexible, broad method for promotion of self-directed learning, formative and summative learner evaluation and competency assessment Planning and implementation and are key Promote research across disciplines to determine best activities and methods of evaluation Resources are available

The Forest

References Challis M. AMEE Medical Education Guide No. 11(revised): Portfolio-Based Learning and Assessment in Medical Education. Medical Teacher 1999; 21(4). Friedman BD et al. AMEE Medical Education Guide No. 24: Portfolios as a method of student assessment. Medical Teacher 2001; 23(6). HC Barrett University of Anchorage

Feedback BLUE FORM Session #: L67A Lead Presenter: Potts

MyFolio Decision Functionality –“Repository for students, residents, fellows, and lifetime learners to collect detailed information about their learning experiences” Input of information by individuals and or administrators Uploading of documents Importation of data from E*Value Data portability Program reporting Flexibility –Unique set up for every client

MyFolio within E*Value Access –Administrators, faculty, and residents already using E*Value Integration –Direct feed from E*Value biographic data –Direct feed from E*Value automated reports

MyFolio Design Process Portfolio Goals: –Learning through assessment –Collaboration of advisor and resident –Documentation of progress towards competency Portfolio Design: –Mixed Model (learner and educator driven) –Competency Based Organizational Scheme

Portfolio Contents The experience The learning The evidence Learning needs Learning opportunities

Competency Based Organization Spreadsheet developed to organize portfolio modulesSpreadsheet –File Organization (Title) –Data –Who will enter –Who will have access

Planting the Seed inviting residents to participate Artist portfolios—how can I apply this to education?

Electronic Learning Portfolio: The Appeal Accomplishments all in one place Presentations organized (not on multiple hard drives) Incentive to update my CV Increased familiarity with the ACGME core competencies Awareness of my strengths and weaknesses Reflections on residency experiences Foundation for discussions with my advisor

Building a Portfolio Initial step: gather the materials to be entered (presentations, CV, etc.) Alternatively, one could start from this point forward Next, enter reflections (mostly ideas I had been thinking about anyway) Continue to maintain an up-to-date portfolio

Experiential Learning Kolb Learning Cycle Kolb D.A. (1984) 'Experiential Learning experience as a source of learning and development', New Jersey: Prentice Hall Private Learning Process Observed Behaviors

Our Evaluation System Web-based comprehensive medical education evaluation system Evaluations include ACGME general competencies in templates that can be easily customized to specific rotations Additionally tracks procedures, conference attendance