1. Join the conversation! Our Twitter hashtag is MSE12 The Reality of Virtual Patient Cases: Four Family Medicine Clerkships’ Experiences Integrating fmCASES

2. Educational Goals and Objectives: Participants will be able to: Describe four schools' approaches to using fmCASES. Discuss positive and negative aspects to different implementation strategies. Imagine new ways to use fmCASES at one's own institution.

3. Introduction(5 minutes) Demonstrations of applications of the fmCASES – 60 minutes: 10 min each with 5 min for questions Group Discussion (25 minutes) Schedule

4. Alpert Medical School of Brown University – Paul George, MD – David Anthony, MD, MSc Medical College of Wisconsin – Douglas J. Bower, M.D. – Karen Hulbert, MD University of Oklahoma – Rachel Franklin, MD University of Vermont – Martha Seagrave, PA-C, BSN Presenters

5. fmCASES Overview Development: 2008-2010 33 virtual patient cases – 29 original cases – 4 new cases in pilot testing currently fmCASES subscription includes 5 CLIPP cases and 2 SIMPLE cases Comprehensive coverage of STFM FM Curriculum (4C) Peer-reviewed/updated annually

6. Subscriptions and Case Use Current Subscribing Schools: 84 Number of Case Sessions: – 2010-2011 (First Subscription Year): 148,209 completed case sessions – 2011-2012 235,791 completed case sessions

7. Clerkship in Family Medicine Alpert Medical School of Brown University

8. Clerkship in Family Medicine Six weeks Didactics at Brown every Wednesday – Lectures, Skills workshops, Case Discussions, PCMH project, Journal Club, Reflective Writing Discussions – Silva Family Cases Simulated Family taught in small groups Readings and facilitation questions Focus on communication and vulnerable populations

9. Lack of a validated exam – We were using the modified Sloane exam Students didn’t “know how to study for the final” Variable clinical experiences – OB or not, rich in Peds or not, etc. Didactics content sometimes redundant with IM and Peds Clerkships Issues seen as of 2010-2011

10. Goal 1: To increase the number of graduating medical students who intend to practice primary care in underserved communities Goal 2: To prepare 100% of AMS graduates to care for underserved patients regardless of their specialty choice Goal 3: Evaluate the impact of Goals 1 and 2 HRSA Grant

11. Integrating the Cases Rewrite of the Silva Paper Cases – Remove the dry medical content from discussion – Link 1-2 fmCASES as prior reading to each paper case – Add one extra reading on “Grey area” Shared decision making, health literacy, options counseling, etc.

12. Integrating the Cases All other lectures have fmCASES linked as assigned reading In all, 24 cases “Required” as prior reading But examined on all 40!!! – Using the validated exam

13. Students know we can track them Open discussion about it at mid-clerkship meeting – I have reviewed the Log Data beforehand – Opportunity for coaching on learning skills No direct penalty for not completing the “Required” cases – Natural consequences Tracking their use

14. 49 students have complete the new Clerkship – Completed Cases Average30.3 Median34 Minimum6 – 11 students complete fewer than the “required” 24 Outcome to date – Use data

15. Prior exam (80 item Sloane exam) in 2010-2011 – Average Score - 73.12% – Standard Deviation - 7.06% (N=106) – Max/Min92%/44.6% fmCASES exam – Average Score-76% – Standard Deviation - 6.73% (N=49) – Max/Min - 87.3%/58.7% Final Exam Data

16. Comparing students completing 24 or fewer cases (N=13) to those completing 25 or more (N=36) <=2425+ Average72.5%77.3% Std Dev6.8%6.2% P-Value for difference = 0.02 95% CI for difference = (0.09 – 1.3) Use of Cases and Exam Performance

17. Student Reaction to fmCASES Generally positive response from students: “They are an outstanding learning tool - relevant, presented in a format where you are likely to remember the material, and rich with additional resources. Every clerkship should use a format analogous to this.” “The FM Cases were extremely helpful in developing not only a solid knowledge base but also gave me the tools on how to approach clinical situations.” “The new FM cases were much better than a text book.” BUT ALSO “My concern regarding them is that they are a bit stifling in the sense that they don't embrace every learning style.” “Long.”

18. Discussion Future Family Doctors

19. MCW Experience with fmCASES/MedU A Six Month Progress Report Douglas J. Bower, MD FM Clerkship Director Karen Hulbert, MD Early Clinical Experience Director Associate FM Clerkship Director Medical College of Wisconsin Milwaukee, WI February 4, 2012 Long Beach, California

20. MCW Experience with fmCASES/MedU A Six Month Progress Report Douglas J. Bower, MD FM Clerkship Director Karen Hulbert, MD Early Clinical Experience Director Associate FM Clerkship Director Medical College of Wisconsin Milwaukee, WI February 4, 2012 Long Beach, California

21. Background:MCW Private Medical College Urban campus in Milwaukee 200 medical students/year M3 Family Medicine Clerkship: 4-week required rotation Family Medicine directed, M1/M2 Longitudinal Early Clinical Experience course

22. Implementing Use of MedU/fmCASES at MCW May 2011 – MCW Learning Resource Grant funded to pay for subscription to MedU for one year: fmCASES, SIMPLE, WISE-MD (Peds already had CLIPP) Proposal submitted by faculty including representatives from family medicine, pediatrics, internal medicine, surgery, MCW Library, basic science, early clinical experience (M1-M2) faculty, M4 course director MedU cases available June 2011

23. Disseminate MedU for use in all 4 years of curriculum – M1/M2 for early clinical experiences, clinical context for basic science. – M3 Clerkships: FM, IM and Surg (Peds already using CLIPP) – M4 Evaluate educational value/usefulness of MedU – Make a case for long-term institutional support for MedU subscription cost, if faculty/student supported Goals for MedU Use at MCW

24. Dissemination/Evaluation of MedU Dissemination – Presentation/Demonstration of Computer Assisted Instruction (CAI) All MCW course/clerkship director meetings (3) One-on-one with select course leaders (pathology, anatomy) Spreadsheet of “topics” for MedU cases – Promoted for use to address ED-2, clinical context for basic science, address national learning objective on M3 year for each specialty Evaluation/Results: 3 Data Sets (DS) DS1: 6-month MedU survey (July-Dec) DS2: MedU database DS3: Individual fmCASES monthly clerkship evaluations

25. N = 148 of 200 students (74%) Q1 = Usage; Q2-6 Likert (SA, A, D, SD); comments Q1 – Have you used MedU cases in last 6 months?* – all students report use (except 2) – fmCASES75 students – SIMPLE76 students – CLIPP116 students – WISE MD6 students M3 year: First 6 months one-half of class on IM, Peds, Surgery; Other half of class on Family Medicine, OB/GYN, Psych, etc. DS1: M3 6-Month MedU Survey July to December, 2011

26. Q2 – Improved my ability to systematically think through a patient’s case – SA = 14%; A = 65%; SA+A = 79% Q3 – MedU enhanced my preparation for Step 2 – SA = 5%; A = 61%; SA+A = 66% Q4 – Effective use of my time relative to other learning modalities – SA = 12%; A = 53%; SA+A = 65% Q5 – Would pay a modest fee to support continued access – Yes = 32%; No = 68% Q6 – Plan to revisit MedU Cases as an M3/M4 student – Yes = 56%; No = 44% DS1: MedU Cases Survey, Likert July to December, 2011

27. Comments : – N = 74 of 148 – 30% Positive; 55% I like them, but …; 15% Negative Positive – “Effectively applies knowledge to a clinical scenario” – “fmCASES … Good supplement to reading” – “I believe MedU cases were overall an excellent way to integrate information during the 3 rd year clerkships” – “Provided a path from diagnosis to treatment to follow up” I like them but … – Provides useful information … but lengthy/time comsuming” – “Refreshing and more interactive way to learn, but less effective as a study/review tool” – … “Would prefer … case files … Q bank, USMLE World, First Aid,” DS1: MedU Cases Survey, Comments July to December, 2011

28. Negative – “They are a slow way to learn. The efficiency is not there.” – “tedious”, too much fluff” – “Didn’t help with prep for the shelf for medicine or pediatrics” – “Knowledge gained … not worth the time” Potential use in M1/M2 year (unsolicited) – “These … would be helpful for M1 and M2 students as part of early medical education” – “Very helpful in walking through a diagnosis... would use more of this teaching method in the first two years.” DS1: MedU Cases Survey, Comments July to December, 2011

29. Integration of fmCASES into M3 FM Clerkship One month FM Clerkship (7, ½ day classroom TBL sessions) 15 assigned cases: – 11 as TBL preparation assignments (2, 5, 6, 8, 17, 18, 21, 25, 27, 29, CLIPP 6) – 4 as “assigned reading” (#3, 20, 28, 33) 170 pages of reading from previous year eliminated (still required: 177 pages of Sloane, 235 pages of Case Files) Evaluation: – MedU database – Survey evaluation of just two cases in real time after TBL (July to December) Case #2 Well Visit Case #25 Shoulder problem

30. Completed Cases (%)Ave. Time per Completed Case* TBL associated assignment 89%56 minutes General assignment48%48 minutes Not assigned, completedN = 20 students (21%); 7 students (9%) multiple cases range 0 to 16 cases/mo., average = 6 cases/mo. 58 minutes DS2: MedU Database (FM Clerkship Students Only) N = 97 students, July to December 2011 * Time per case was +/- 2 minutes July vs. December

31. DS3: M3 FM Clerkship Overall Value to Learning fmCASE 2 & 25 fmCASE #2 N=100fmCASE #25 N=99 5 –Extremely Valuable12%27% 4 –Moderately Valuable65%50% 3 –Slightly Valuable16%20% 2 –No value2%1% 1 –Did not spend significant time on it2% Mean3.84.0 SD.7.9

32. DS3: M3 FM Clerkship fmCASE #2 & #25 Real-Time Eval TBL Prep, Case Specific (Means) Scale: SA-6, A-5, SA-4, SD-3, D-2, DS-1, NA fm #2fm #25 TBL Preparation Appropriate to my level of training5.15.0 Provided important foundational knowledge for this TBL4.95.0 Case Specific Multiple-choice questions useful4.64.7 Network useful---3.8 Expert sections useful4.64.7 Links to additional information useful4.84.7 Summary link was useful---5.1 Other video links useful---5.2

33. DS3: M3 FM Clerkship fmCASE #2 & #25 Real-Time Eval MCW Curriculum (Means) Scale: SA-6, A-5, SA-4, SD-3, D-2, DS-1, NA fm #2fm #25 MCW Curriculum Would recommend to future students4.64.8 More relevant to learning than textbook of same material4.74.8 Could be used in M1 or M2 year*4.64.8 Would favor tuition support for MedU vs. other electronic resources4.94.6 * Per student comment’s, MedU cases could be used in anatomy, pathology, physiology, physical exam course, interview course. Student Comment: “the curriculum would need to become more clinically oriented (to use MedU and M1 or M2 year)”

34. The LEAP Program (Longitudinal Experience Advancing Patient care)  Part of the Pilot Integrated Curriculum (2010-2012)  28 students (now M2’s) have weekly clinical experience with preceptor (1/2 day)  Sites include FM Residency programs, MCW Primary Care clinics (Fam Med & IM), community Pediatric & Internal Med clinics and the VA system  Students keep a patient experience log (minimum # of entries = 30 per semester)

35. Integrating fmCases & Next Steps: Challenge: Variation of sites means variation of experiences/exposures to different patient populations – Assign 4 cases during the M2 year to balance the experiences in their weekly clinics – They can “count” the cases as “patients” in their experience log New “Discovery Curriculum” at MCW approved for entire incoming class (=204) in August 2012 LEAP becomes “Clinical Apprenticeship” course which begins in M1 second semester Variability of sites will increase to accommodate entire class & will include specialists in addition to primary care

36. Impressions / Lessons Learned (based on 6-month experience) 1)fmCASES in FM clerkship useful as preparation for TBL session –fmCASE assignment, not completed as often without direct accountability (e.g., TBL cases) –Students use them, if assigned (average one hour) –Not perceived as “study resource” 2)MedU cases overall perceived as positive learning experience, modestly valuable by students 3)Limitations of MedU usage reports

37. 1)Outcomes of MedU cases on learning not yet assessed (OSCE/MCQ end-of-clerkship exam) 2)Sequence of MedU case  use from M1 → M4? 3)Sustained student acceptance ? 4)iPad Pilot – ongoing 5)Institutional funding??? Questions Looking Forward

38. Discussion

39. Rachel Franklin, MD University of Oklahoma College of Medicine

40. Experience with fmCASES at University of Oklahoma Rachel Franklin, M.D.

41. Background Public University 2 campuses: Oklahoma City and Tulsa, OK Campuses linked but separately administered (Ed-8 challenge) 160 medical students/year (total) Family Medicine: 4 week required rotation (MS-3)

42. Weekly longitudinal Patient-Based Learning session (PBL) Twice-Weekly PBL sessions on acute/chronic care Online quiz bank of hot topics Dermatology lecture Musculoskeletal lectures (3) and skills session Mid-clerkship feedback Objective Clinical Skills Examination (OSCE) Criterion-referenced in-house written examination Prior Curricular Structure

43. OKC: student feedback Tulsa: faculty shortage Pilot program participant and case author (fmCASES 18: Headache) Proposed structure: – Once-weekly sessions (OKC, Tulsa) – Add Note Writing session (OKC) – Add EKG analysis session (Tulsa) – Keep longitudinal patient case (OKC) Another component needed to ensure Ed-2 (recorded via MedHub) Curricular Challenges

44. First: curricular review: – Can we replace PBL with fmCASES? – Can we maintain faculty feedback/teaching opportunities? 22 cases required – Week 1: Preventive Care and Musculoskeletal Pain (cases 1,2,4,10,11) – Week 2: Chronic Disease and Complications (cases 6,7,8,9,13,18) – Week 3: Assessing Undifferentiated Complaints (cases 3,19,20,21,23,24,26) – Week 4: Reproductive Health and Dermatologic Lesions (cases 14,16,17,27) Integrating fmCASES

45. Weekly longitudinal Patient-Based Learning session (PBL) Weekly fmCASES Quiz and Faculty-led feedback session Dermatology lecture Musculoskeletal lectures (3) and skills session Weekly Progress Note presentation and feedback session Mid-clerkship feedback Objective Clinical Skills Examination (OSCE) Current Structure

46. Weekly quiz review – 10 questions, multiple choice – Worth 2% of final grade each week – Faculty-led feedback session Final exam – 100 questions – fmCASES exam, 4-5 questions per case – Worth 40% of final grade fmCASES-Supported Curriculum

47. Standardized curricular component – Guaranteed remediation of Ed-2 – How we track in MedHub – OB/STD most often not seen in “real” patients Fewer classroom sessions – OKC students happy not to have 2X/week – Tulsa faculty able to cover feedback Improved efficiency of classroom sessions – Added EKG analysis PBL to longitudinal patient – Added Progress Note assignment and feedback sessions Benefits

48. LCME Ed-2 – caution advised! Cases no longer being used as intended – Too much available in PDF files? – Time per case year 1: 20-120 minutes – 5 minute minimum rule for OU – Campus print shop Drawbacks

49. Discussion Future Family Doctors

50. The University Of Vermont College of Medicine Martha Seagrave PA-C,BSN

51. UVM Clerkship in Family Medicine Six week clerkship starting March 2011 1 week orientation Skills sessions, standardized patient experiences, lectures/discussion 5 week clinical experience Community Health Improvement Project fmCases and exam

52. Clinical Experience Clinical sites in Vermont, Maine and Connecticut 75% in Vermont the rest in Maine and Connecticut farthest site 9+ hours drive 60+ different preceptor sites Over 100 different preceptors

53. Benefits – Increased diversity – More choice increasing quality of options – Less burnout Challenges – Maintaining comparable experiences – Monitoring quality

54. “Family Medicine Bridge Week” – FM Clerkship received highest ratings from students Midclerkship meetings with UVM COM clerkship team fmCASES Response

55. 20 preselected cases required, all encouraged 7 cases directly related to topics covered in skills week Other MedU cases used at UVM include CLIPP students choose 5 out of 8 recommended cases (32 total) WiseMD cases are optional SIMPLE and CORE not utilized at this time Students have access to all cases--purchased by COM fmCASES at UVM

56. Elimination factors – Textbook – UVM generated CAI – Textbook associated exam

57. fmCASES exam 75 questions, 2 hours – Questions associated with identified cases, selected from FEQ database – All questions grouped by case—~ 10 questions available per case, 3-4 each chosen for UVM exam Outcomes – More challenging for students than previous exam – It is now the determining factor—honors/pass

60. Benefits Consistent curriculum over large geographic area More active learning experience Encourages clinical reasoning Models the effective preceptor

61. Challenges Which cases to include Inconsistent internet access at rural sites “Make your own” exam option, no current way to benchmark results against other schools/shelf exam Time to complete cases

62. Student response “I had completed the fatigue case and the next day I had a patient who presented with a chief complaint of fatigue. I felt well prepared.” Take a lot of time to complete. “The cases simulate real practice, much more effective rather than reading pages in a text book.” “Why can’t we just have the PDF’s without doing the cases?”

63. Conclusions – Curriculum is up to date – Exam is up to date – Curriculum is consistent across sites – Student satisfaction is generally positive

64. Innovations – Used for FM resident remediation – Working with academic and community faculty CME Evidence-based resource Familiarity with student curriculum

65. Discussion Future Family Doctors

66. Thank You For Coming!