1. Portfolio Assessment in Clerkship Michelle Gibson - Geriatrics (thanks to Chris Frank and Melissa Andrew too)

2. Objective Participants will discuss the use of portfolios in assessing medical trainees (after a brief review of some basic principles…)

3. Background Change to the Internal Medicine clerkship in January 2007 4 weeks to 6 weeks Opportune time to review the assessment process in geriatrics Previously, 8 assessment cards filled out by MD supervisors

4. Challenges with old system Cards were often filled out the last week of the rotation Depended solely on supervisor impression No direct observation was incorporated Students were assessed using only one modality

5. Reasons for change Desire for more formative assessment opportunities Desire for more robust assessment (Need to meet course requirements!)

6. Formative assessment Assessment that occurs during instruction to provide feedback to teachers and students Describes needs for future learning Assessment for learning

7. Summative Assessment Assessment that occurs at the end of an instructional unit to document student learning Certifies learning Assessment of learning Is this portfolio formative or summative?

8. High Quality Assessment Clear & appropriate learning targets Appropriateness of assessment methods* Validity Reliability Fairness Positive consequences* Alignment* Practicality & efficiency*

9. Reliability & Validity Reliability: concerned with the consistency, stability, and dependability of the scores Validity: the appropriateness of the inferences, uses, and consequences of the assessment Is it assessing what you want it to assess?

10. Portfolio A systematic collection of student products to assess progress Structured portfolio The assessor determines the mandatory content of the portfolio

11. Literature review Assessment of housestaff in clinical rotations Portfolio literature reviewed Specific instruments examined Portfolio components were selected to provide multiple methods of assessment for each clerkship objective

12. Components by Methodology MethodologyPortfolio components Foundational evaluationsAssessment cards - 6 Direct observationsMini-CEX - 2 Practice- & Data-based learning Medication Review Self-assessment of case write-up Multi-source evaluationsTAB - 2 Miscellaneous (optional)Student’s choice

13. Mini-CEX

14. TAB

15. Card

16. Medication Review - written Based on an actual patient, preferably one for whom the student has responsibility brief summary of patient, meds he/she is taking including whether or not they are appropriate suggestion of other medications he/she should be taking

17. Self-assessment of write-up H & P from early in rotation Students asked to read and identify 2 things done well 2 things that could be improved Reviewed with undergrad coordinator on the last day

18. Components & CPE Categories CPE CategoryECMCTABMRCase Knowledge BaseXXXX Motivation to LearnXXXXX Team FunctioningXX CommunicationXXXXX Relationship with patients/familiesXXX Resource ManagementXXXX

19. Process Orientation with students on Day 1 Team, faculty, & residents are oriented to the portfolio Students meet with undergrad coordinator at mid-term, review progress, identify any gaps, etc. Last day - review portfolio with students, completion of CPE

20. Evaluation of Pilot Focus on acceptability Students and assessors Student evaluations ongoing - not seen by the undergrad coordinator until collated in April 2008 Will redo faculty portion in December 2007

21. Challenges noted - 1st time First rotation ever for clerks Time spent learning how to be a clerk Couldn’t assemble half the portfolio in 3 weeks Exam stress “detracted from our learning” Supervisors were absent more than usual *all except exam stress have resolved

22. Strengths noted Multidisciplinary team members provided high-quality feedback Direct observation occurred Forms were felt to be user-friendly, acceptable Students appreciated discussing their medication review and being observed

23. Assessor ratings of forms TAB 5/5 for ease of use; 5/5 for usefulness Mini-CEX 5/5 for ease of use; 4/5 for usefulness Time spent by assessor: 13.5 minutes Encounter card 4.5/5 for ease of use; 4.5/5 for usefulness Time spent by assessor: 15 minutes

24. My time? 5-10 minutes on day 1 for orientation Plan to develop “talking power point” 15 min per student at the end of week 3 Mid-term review Could be any faculty 15 min per student on the last day Final evaluation Could be any faculty

25. My *impressions* Students receive more feedback in a timely fashion Direct observation doesn’t take long, and is very valuable TAB form is very useful, and team members are pleased to be filling them out Mid-term meetings are very useful

26. Future directions Continue with student and assessor feedback Incorporate other tools? Peer assessment? Patient assessment? Study it more Implement it for residents

27. Factors to note Small unit, discrete group of nursing and other staff Small division Very workable for us … but for others?

28. Questions to consider Is this high quality assessment? How could it be improved? Can it be extended into other areas? Is there a role for a “whole clerkship” portfolio? “Pre-clerkship” portfolio? Residents & non-MDs as assessors? Electronic?

29. Comments about our students “.. Is very attentive… he listens well to pt. & staff, & then goes off & evaluates what he’s been told” “excellent verbal communication observed with all team members” “better H & P than most residents”