Online, Self-Paced Learning Modules in the Family Medicine Residency A Pilot Intervention Eric Skye MD, Tara Master-Hunter MD, Leslie Wimsatt PhD, Elizabeth Wilson MFA Department of Family Medicine University of Michigan

Initial Project Goals Increase faculty skills in designing, implementing & evaluating web-based modules Provide online learning opportunities that allow for independent & individualized learning Extend resident knowledge, skills and attitudes with regard to core curricular content Address & document progress toward competencies Provide tools for resident self-assessment and improvement

Background/Context Past research Department considerations Examples  Online learning is creating a paradigm shift Online modules require an investment of time Little is known about affect on faculty developers or resident learners Examples  3 “generations” of online modules * Department considerations Size/scope of curriculum + scheduling constraints Increased demands on residency education Limited effectiveness of isolated conferences & electives Past research Transition to online learning is creating a paradigm shift in teaching and learning Online modules require considerable time to design, pilot test and manage Little is known about how development of web-based curricula affects faculty and resident learners Department considerations Size/scope of curriculum + scheduling constraints Increased demands on residency education linked to changes in national standards Need for training in patient use of alternative therapies Limited effectiveness of isolated conferences & electives

Methods Study population (N=76) Department of Family Medicine, University of Michigan 47 faculty/staff members 29 resident learners Data Collection Online survey instruments Demographic information Closed and open-ended response items Pre-/post quizzes Secondary analysis Quantitative data Frequency distributions, descriptive statistics, paired t-tests Qualitative data Thematic analysis of participant feedback 47 faculty/staff members 6 work team coordinators 26 faculty authors 15 faculty peer reviewers Demographic Data Faculty/staff participants – Core work team members + faculty authors Professional job title Project role Prior experience with instructional technology Level of training or support received Family Medicine residents – Unit evaluators Residency year Basic computer/Internet expertise Prior experience with online module use SURVEYS… Five-point Likert scales Perceived importance Agreement; satisfaction Difficulty of use

Web-Based Surveys Process data – core project team & faculty authors Online module feedback – residents, faculty and staff Professional development reflections – faculty authors Outcome measures Participant satisfaction Software usability and functionality testing Faculty workload Support/service adequacy Affect of module work on approaches to teaching, learning and/or assessment Process data – core project team & faculty authors Satisfaction with work progress Perceptions regarding project experience Level of training + support received Effectiveness of work processes and support received Online module feedback – residents, faculty and staff Usability testing & expert review Ratings  technology functionality + design Content review Professional development reflections – faculty authors Self-assessments Affect of module work on approach to teaching, learning and/or evaluation

Pre-Post Quizzes Online module pre/post-test data – resident learners Quiz questions 3 to 10 content-specific items per quiz Multiple choice, matching, rank order formats Also case vignettes + self-evaluation items (non-scored) Linked to learning objectives + ACGME competencies Peer and expert reviewed Pilot tested Outcome measures Content knowledge assessment

Educational Intervention Development + pilot testing of 10 web-based modules with 43 embedded sub-units Traditional family practice (6 modules) Musculoskeletal examinations Obstetrical procedures Women’s health Critical care Clinical procedures Practice Management Integrative medicine (3 modules) Integrative approaches to disease Mind-body medicine Body-based therapies Educational best practices (1 module)

Results: Faculty Author Feedback (N=15) Satisfaction or dissatisfaction with… (5-point scale)1 Mean SD Orientation to the project & unit development process 4.4 0.65 Training for creating learning objectives 4.3 0.87 Training for determining and planning content 4.1 1.03 Development of text materials 4.0 0.50 Peer review process 0.98 Designing pre-/post-test questions 3.9 0.77 Proposal forms, author checklist, timelines provided 0.57 Piloting process 3.7 Training for storyboarding 3.6 0.95 Designing for the web-based format 3.5 0.82 Satisfaction or dissatisfaction with the module development process… 1 Response scale coded: 1=Very dissatisfied, 2=Dissatisfied, 3=Neutral, 4=Satisfied, 5=Very satisfied.

Results: Faculty Author Feedback (N=15) Ease or difficulty with … (5-point scale)1 Mean SD Developing content material 3.9 0.33 Creating learning objectives 3.8 0.79 Designing pre-/post quiz questions 3.3 0.91 Organizing/structuring online content 3.1 0.99 Meeting recommended timeline check points 2.6 0.93 Software use 2.3 Scheduling time to work 2.0 0.74 Agree or disagree… (4-point scale)2 Paid time allotted for training was appropriate 2.9 0.57 Paid time allotted for development was appropriate 0.66 1 Response scale coded: 1=Very difficult, 2=Somewhat difficult, 3=Neutral, 4=Somewhat easy, 5=Very easy. 2 Response scale coded: 1=Strongly disagree, 2=Disagree, 3=Agree, 4=Agree strongly.

Results: Resident Usability Testing (N=103)* Percent (%) *N=Aggregated responses

Results: Resident Pre-/Post Test Scores p<.05*, p<.01**, p<.001***

Results: Pre-/Post Test Scores (cont.) p<.05*, p<.01**, p<.001***

Results: Faculty Comments Learner needs “Rather than thinking “what do these students need to know?,” the project put “how will these students be using this information?” front and center.” Creating the unit helped me be more specific about various steps in the learning process and helped me see the unit from the learner’s perspective.” Content dimensionality “I had always thought about [content] in terms of an outline or PowerPoint, not in terms of the forward, backward and sideways of a web-based approach…” “As a linear thinker, I found this project to be particularly challenging, and rewarding. Teaching pedagogy “[This] work taught me to address the differences in learner style…to pace my instruction and avoid over delivery of information. This process has broadened my understanding of teaching methodologies.”

Discussion Study contributes an understanding of opportunities and limitations in adding web-based curriculum to a residency Limited faculty development required for online material development Online material generally well-accepted by residents Improvements in testable knowledge demonstrated Clinical knowledge/skill development = unknown Optimal volume of web-based curriculum remains unknown

Discussion (cont…) Considerations for curriculum planning Work hours restrictions On-duty vs. off-duty requirements Unique strengths Customized curriculum Available anytime, anyplace Sequential preparation and teaching Delivery to off-site or other distant learners Accountability/burden on program Optimal volume of online teaching

Future Research Impact of online vs. group learning on future family physicians Validation of improvement in clinical care/skills through web-based curriculum Identification of optimal faculty development and support processes

Online, Self-Paced Learning Modules in the Family Medicine Residency For additional information, contact: Eric Skye MD eskye@med.umich.edu