Extending Use of Assessment Data to Support Long-term Curricular Improvement Leslie Wimsatt Ph.D., James Cooke M.D., Eric Skye M.D. Department of Family Medicine University of Michigan

Disclosures No disclosures or conflicts of interest

Background/Context Department of Family Medicine: Online module curriculum grew from 19 to 31 modules in 3 years Pre-/post-tests showed statistical improvement Positive resident evaluations (value, satisfaction, relevance) Best fit between module content educational goals and day-to-day patient care needs remained unclear Study goal: To pilot an assessment protocol and examine its utility for use in ongoing curricular review Online modules offer a way to overcome barriers to educational delivery Advantages... Accessibility Economies of scale Flexible scheduling Ease in updating Individualized instruction Incorporation of novel instructional methods Successful management requires… Commitment to measuring results based on meaningful criteria Using data to inform curriculum evaluation Initiating change to improve learning results Rosenberg M. E-learning: strategies for delivering knowledge in the digital age. NY: McGraw-Hill; 2001. Cook D. Where are we with Web-based learning in medical education? Med Teach 2006;28(7):594-598.

Methods Study population (N= 30 residents) Department of Family Medicine, University of Michigan Data Collection Two focus group discussions (4 questions w/probes) Modules completed to date Application of module learning to clinical practice Potential for effectively using other online tools Module testing exercises (6 modules) Content-specific post-post testing (7-17 items per module) FSE/IUPC Hip Exam Knee Exam Joint Injection Skin Surgery Vasectomy

Methods (cont.) Data Analysis Qualitative data Quantitative data Thematic analysis of participant feedback Single author review Quantitative data Frequency distributions Descriptive statistics Paired t-tests

Results Response rate Content analysis – pre-/post-testing 73% (22/30) Content analysis – pre-/post-testing Significant improvement in pre-/post knowledge test scores for all 6 modules previously tested (p<.05) * Post-post testing exercise Significant improvement in pre-/post-post knowledge scores for 4 of 6 modules Joint injection (p=.003), (p=.Knee (p=.049), Skin Surgery (p=.035), Vasectomy (p=.028) Knowledge scores fell significantly across most modules (post-/post-post) *Skye EP, Wimsatt LA, Master-Hunter TA, Locke AB. Developing Online Learning Modules in a Family Medicine Residency. Family Medicine. 2011;43(3):185-92

Among residents who took all three tests – pre, post, and post-post…   Pre Post-Post p= Post FSE-IUPC (N=7) 66.8 (20.43) 85.6 (4.73) .054 86.15 (8.38) .916 Joint Injection (N=8) 71.9 (19.39) 81.8 (18.07) .250 97.4 (2.17) .041 Knee (N=10) 65.5 (14.31) 81.1 (18.20) .044 100.0 (0.00) (18.2) .009 Hip (N=7) One very low score, HOI 55.2 (19.55) 63.1 (17.24) .449 99.4 (1.59) .002 Skin Surgery (N=11) 71.8 (17.57) 82.8 (10.02) .068 93.4 (10.39) .060 Vasectomy (N=8) 2 lower scores HO2, HO3 50.6 (16.22) 77.1 (20.34) .033 97.5 (4.63) .040 Among residents who took all three tests – pre, post, and post-post… Column 2-4: Pre- and Post-Post Test Results Column 6-8: Post- and Post-Post Test Results

Results (cont.) Experience with online module curriculum All resident participants - 22/22 How/what information from online modules is applied in clinic Procedural skill modules = most beneficial/transferrable knowledge Knowledge/content-heavy modules = least transferrable information Video provides highly transferrable information Access to modules in clinic = an essential element How to improve access to module content in clinic Centralized web portal = essential Module learning most frequently applied to patient care All procedural modules Specifically mentioned all musculoskeletal modules, skin surgery, IUD insertion

Results (cont.) Online modules vs. lecture material … “If I’m going back to a module I know exactly what I want to access and I know where it is because I’ve already clicked through it… as opposed to a lecture where it’s just kind of there and I’m not sure of the order of the slides… it just seems like if I go back to reference a module, it’s faster. ”

Results (cont.) Ability to sort, control, manipulate material… “I think pacing is the biggest thing. Like in a lecture, some lectures are too slow for you. Some lectures are too fast. The control is the biggest thing. Really, if you have a part that you are weaker at, you can spend a lot of time and maybe look up an additional resource on the same side or on the same screen. I do that all the time for things that I’m not so good at, whereas things that I’m better at, I can sort of power through a bit more.” Ability to sort, control, manipulate material…

Limitations Small sample size limited subgroup comparison of knowledge gains Focus groups were semi-structured, thereby confining resident comments to targeted areas of questioning Single program experience limits generalizability

Results (cont.) Active use during clinic… “I use it when I [have] somebody on the schedule … with knee pain or something, I’ll go look at the video real quick … or remind myself of the maneuvers and the names.” “There’s a document on those that tells you how to document the exam, so you can read through it while you are dictating. I’ve done it that way too.”

Results (cont.) “I think that one of the benefits of the modules is that [they are] mixed media so you can use to prepare for something ahead of time, you can use it in the clinic at that moment, even if it is something that you kind of know but you want to document. It is kind of a resource on multiple levels.”

Discussion Post-Post testing protocol provided a method to assess resident knowledge Resident approaches to module use vary Residents value the organizational framework that some modules provide Ease of access and use emerged as essential elements

Future Directions Protocol will be incorporated into future curriculum assessment efforts Results will be used to inform module content revisions, enhance integration of module use throughout the department, contribute to improvements in faculty precepting, and expand application of module learning to patient care Additional research needed to fully identify long-term differences in learning retention

Open Access Modules Musculoskeletal Medicine Women’s Health http://sitemaker.umich.edu/fm_musculoskeletal_knee/introduction http://sitemaker.umich.edu/fm_musculoskeletal_hand-wrist/introduction http://sitemaker.umich.edu/fm_musculoskeletal_shoulder/introduction http://sitemaker.umich.edu/fm_gmeig_musculoskeletal_joint-inject-aspir/musculoskeletal_module_-_joint_injection_unit Women’s Health http://sitemaker.umich.edu/fm_womens-health_iud/introduction http://sitemaker.umich.edu/fm_womens-health_endometrialbiopsy/introduction http://sitemaker.umich.edu/fm_gmeig_womens-health_osteoporosis-prevention/introduction___directions http://sitemaker.umich.edu/fm_gmeig_integrative-approach-to-menopause/introduction

Open Access Modules Pain Management Integrated Medicine Modules http://sitemaker.umich.edu/fm_gmeig_palliative-care_pain-management-nonopiates/introduction http://sitemaker.umich.edu/fm_gmeig_palliative-care_pain-management-opiates/introduction Integrated Medicine Modules http://sitemaker.umich.edu/fm_gmeig_integrative-approach-to-disease_gerd/introduction http://sitemaker.umich.edu/fm_gmeig_integrative-approach_diabetes/introduction http://sitemaker.umich.edu/fm_integrative-approach_asthma/introduction http://sitemaker.umich.edu/fm_gmeig_integrative-approach-to-menopause/introduction

Extending Use of Assessment Data to Support Long-term Curricular Improvement For additional information, contact: Leslie Wimsatt, Ph.D. lwimsatt@med.umich.edu