1. Web-Based Portfolios: A Powerful New Tool for Surgical Education Catherine E. Lewis MD, Areti Tillou MD, Michael W. Yeh MD, Chi Quach, Jonathan R. Hiatt MD, O. Joe Hines MD Department of Surgery David Geffen School of Medicine at UCLA

2. Professional Competence Competency-based model for GME - 1998Competency-based model for GME - 1998 “Habit of lifelong learning”“Habit of lifelong learning” – Leach “Reflection in daily practice for benefit of“Reflection in daily practice for benefit of individual and community” individual and community” – Epstein & Hundert Surgical educators challenged toSurgical educators challenged to teach, assess and document competence teach, assess and document competence

3. Portfolios: Multifunctional tool for competence Documentation:Documentation: Collection of resident’s workCollection of resident’s work Demonstrate learning, achievement, deficiencyDemonstrate learning, achievement, deficiency Assessment:Assessment: Reliable and validReliable and valid Requires well-defined criteria & structureRequires well-defined criteria & structure Teaching and development:Teaching and development: Guided by mentorsGuided by mentors Promotes self-assessment and growthPromotes self-assessment and growth

4. Portfolio Models ModelContent Benefits & Uses Learning (Growth) (Growth)FlexibleSelf-assessmentMentoring Lifelong learning Assessment (Showcase) (Showcase)Structured Reliable & valid assessment Professional certification HybridBoth Use for both formative & summative assessment summative assessment facilitated by content facilitated by content access privileges access privileges

5. Specific Aims Describe UCLA portfolio systemDescribe UCLA portfolio system Analyze effects of the portfolio systemAnalyze effects of the portfolio system Survey of residentsSurvey of residents Evaluations by facultyEvaluations by faculty Portfolios in Surgical Education

6. UCLA Portfolio System Learning modelLearning model Web-based, password-protectedWeb-based, password-protected 2 GB / user2 GB / user Developed by expert groupDeveloped by expert group Development cost $25,000Development cost $25,000 Launched May 2007Launched May 2007 Portfolios in Surgical Education

7. Content Confidential Personal Public Letters of recPresentations Incid. reportsPublicationsReadings DisciplineCVGoals / objs Eval reports Licensing Portfolios in Surgical Education

8. Content Confidential Personal Public Letters of recPresentations Incid. reportsPublicationsReadings DisciplineCVGoals / objs Eval reports Licensing Portfolios in Surgical Education

9. Content Confidential Personal Public Letters of recPresentations Incid. reportsPublicationsReadings DisciplineCVGoals / objs Eval reports Licensing Portfolios in Surgical Education

10. Content Confidential Personal Public Letters of recPresentations Incid. reportsPublicationsReadings DisciplineCVGoals / objs Eval reports Licensing Portfolios in Surgical Education

16. Surgery Education Portfolio

17. Portfolios in Surgical Education

18. Resident Survey Design ComprehensiveComprehensive VoluntaryVoluntary MeasuresMeasures Understanding of systemUnderstanding of system AcceptanceAcceptance UseUse Perceived effect on competenciesPerceived effect on competencies Response rate 69%Response rate 69% Portfolios in Surgical Education

19. Resident Survey Results Portfolios in Surgical Education % Agree Appreciate organizational function 100 Purpose / functions explained 95 View as important education tool 87 Understand uses of system 82 Aided development of competence 45

20. Faculty Evaluations Comparison pre- and post-portfoliosComparison pre- and post-portfolios Global assessmentsGlobal assessments End-of-rotationEnd-of-rotation Competency basedCompetency based Capture, n:Capture, n: Before implementation = 1488Before implementation = 1488 After = 697After = 697 Portfolios in Surgical Education

21. Improvement in All Competencies Portfolios in Surgical Education Medical Knowledge Patient Care Professionalism Inter- personal Skills Practice Based Learning Systems Based Practice Pre Pre Post Post

22. Patient Care Portfolios in Surgical Education PrePostp Obtains complete H & P4.184.240.04 Generates appropriate ddx4.104.141.16 Develops appropriate rx plan4.084.150.02 Provides appropriate patient care4.244.250.38 Adequate operative prep of pts4.184.230.07

23. Medical Knowledge Portfolios in Surgical Education PrePostp Fund of basic & clinical knowledge4.074.120.07 Engages in ongoing learning4.004.050.10 Prepares for operation4.074.120.09 Understands risks and benefits4.114.190.01

24. Professionalism Portfolios in Surgical Education PrePostp Follows through on tasks4.344.320.71 Demonstrates integrity4.444.450.38 Well groomed, appropriate attire4.434.440.38 Available to patients/family/faculty4.374.390.28 Communicates w/ attending effectively4.334.370.13 Knows limitations4.254.310.05

25. Portfolios in Surgical Education PrePostp Demonstrates care & concern4.354.380.18 Communicates effectively: With patients4.314.350.13 With other healthcare providers4.274.320.08 Maintains medical record4.284.330.07 Interpersonal and Communication Skills

26. Systems Based Practice Portfolios in Surgical Education PrePostp Provides cost-conscious care4.144.170.21 Effectively uses system resources4.204.230.20

27. Portfolios in Surgical Education Reflect critically Aptitude for change Teach others Overall PBLI Practice Based Learning & Improvement

28. Portfolio Learning Helix Portfolios in Surgical Education Experiences, achievements and deficiencies recorded in portfolio Resident reflects in order to learn and draw meaning New knowledge is tested on new experiences which then are recorded PBLI Mentoring

29. Keys to Portfolio Success MentorshipMentorship Support and acceptance by usersSupport and acceptance by users Minimal time commitmentMinimal time commitment Portfolios in Surgical Education

30. Limitations of Present Study Early analysis after implementationEarly analysis after implementation Other improvements in residencyOther improvements in residency Portfolios in Surgical Education

31. Conclusions Portfolios can aid in developmentPortfolios can aid in development and documentation of competence and documentation of competence Favorable response by residentsFavorable response by residents Positive effect for all competenciesPositive effect for all competencies Valuable organizational toolValuable organizational tool Integral to lifelong learning andIntegral to lifelong learning and professional development professional development Portfolios in Surgical Education