1. Initial Outcomes of a Reflective Portfolio and Advising System for Clinical Medical Students Scott Moser, MD Alan Blakely, PhD Anne Walling MB, ChB

2. Why a Portfolio Project? Make the competencies real and known Assess 4 specific competencies (LCME) Provide a practical advising system (linked to competencies) Learn across clerkship “silos” Address the “hidden curriculum” Get added value from work projects Prepare students for lifelong recertification

3. Specific Graduation Competencies Addressed by Portfolio –Develops reflective practice habits, using analysis of experience to improve performance. (Practice-Based Learning and Improvement) –Outlines strategies to continuously update personal clinical knowledge and skills. (Practice-Based Learning and Improvement) –Accepts constructive feedback as part of a commitment to continuous learning and improvement. (Professionalism) –Recognizes and addresses personal limitations, attributes or behaviors that might affect effectiveness as a physician. (Professionalism) Advisors provide assessments of these in MSPE (“Dean’s Letter”) at end of project

4. Key Features Advisors = physicians from all clinical departments Advisors ≠ clerkship directors, dept chairs, or deans (semi-permeable membrane) Each advisor assigned 1-2 MS3s without regard to student career interest 3 meetings from fall of year 3 through fall of year 4 Each meeting focused on 2-3 competencies in developmental order Students collect “artifacts” in e-portfolios prior to each meeting, primarily “reflective” rather than “new” work Advisors submit brief written report on 4 competencies to dean’s office

5. 1. Student preparation CPR Self-Evaluation Case 1 and comments* Case 2 and comments Institutional preparation CPR Faculty Evaluation 2. Student preparation CPR Self-Evaluation SP video and SOAP Note and comments** Institutional preparation CPR Faculty Evaluation 3. Student preparation CPR Self-Evaluation Case 1 and comments*** Case 2 and comments Institutional preparation CPR Faculty Evaluation Patient Care Medical Knowledge Practice-Based Learning and Improvement Interpersonal and Communication skills Professionalism Systems-Based Practice Dean’s Letter Input Published Content (e.g. for Residency) *Meeting 1: Two case write-ups a) one focusing on chronic care/behavioral issues b) one focusing on acute care issues in which the patient presented with an undifferentiated problem and in which the student can demonstrate thought and/or the decision process **Meeting 2: One standardized patient “package” – the video and SOAP note ***Meeting 3: Two case write-ups a) one focusing on ethical issues b) one in which multiple specialties or interdisciplinary cooperation was necessary. Focus should be on enhancing collaboration and patient safety. Learning Contract Ongoing Issues Advisors notes Learning Contract Ongoing Issues Advisor notes Before MeetingMeeting DiscussionOutcomes

6. Evaluation Survey of student and advisor perceptions of advising and professional development Change in knowledge of competencies, advising, revisiting of work Student focus groups at mid-point for reassessment Survey of workload impact, practical implementation Search for connection to objective measurements (Grades, exam scores, etc.) [Ongoing work]

7. Student Surveys How many perceived that “there are faculty from whom you could seek advice on developing…” Before (50)After(5) Technical Skills? 92% 100% Professional Performance? 78% 80%

8. Student Surveys “How many of the six School of Medicine Core Competencies can you list?” Before (50)After (5) Patient Care: 2% 60% Medical Knowledge: 8% 40% Practice-Based Learning: 2% 60% Interpersonal & Communication: 2% 80% Professionalism 56% 80% Systems-Based Practice: 0% 40%

9. Student Survey Comments “Certain aspects of the project were well intentioned but…” “I liked the idea of having an advisor and having scheduled meetings with him or her, but the notion of having assignments and documentation prepared was annoying and time consuming” “The project has failed to add anything to my educational experience except how to manage more busy work.”

10. Advisor Surveys Is this an opportunity to…/Does the student…(12/36 respondents) Excellent NeutralMarginal or adequateor poor Develop reflective 91.6% 0% 8.3% practice habits? Outline strategies to 66.7%25% 8.3% continuously improve? Accept constructive feedback? 83.4%8.3% 8.3% Recognize and address personal 83.4%8.3% 8.3% limitations?

11. Advisor Survey Comments “May be more beneficial for the marginal student. For the student making good progress seems to be a lot of work for marginal return.” “My two students are telling me that this is busy work… We did have meaningful discussions together on other issues of career choices, rotations, etc., so I do not think that a mentorship program is bad at all…” “I think my encounters went well because my mentee came very well prepared with her own agenda for discussion.”

12. Workload and Impact Students reported preparation averaged 2-3 hours and ranged from < 1 to 4.5. Advisors reported prep averaged 45 minutes and ranged from “minimal” to 2 hours. 5 staff spent about an hour in initial student/advisor contact; 2 staff spent an additional 40 hours doing coordination, scheduling and distribution of materials per meeting. Key program personnel (staff and faculty) dedicated additional time to design, planning and guidance.

13. Additional Outcomes Identified 1 student in need of academic dean attention 2 students for whom early advisor intervention seemed to avoid need for later dean’s attention Mentoring developed among occasional pairs Advisor attrition: of 36, 1 withdrew, 2 left town, 2 were “clunkers” New advisor recruitment: easier second cycle than first

14. Conclusions In spite of our efforts to keep it simple, the system remained too complex and felt like “busy work” –Dump electronic portfolio for paper –Overhaul 2 nd meeting format We under-estimated personnel needs and training needs –Even the best teachers needed help –Big paradigm shift for all Evaluation procedures broke down Students and advisors struggle with reflective practice Advisor piece more important than Portfolio piece

15. Future Portfolio/advisor system is “normal” KISSer and train more Expand to both campuses