1. Mary Edwards, MLIS, AHIP, EdD; Erik Black, PhD Health Science Center Libraries and UF College of Medicine, Department of Pediatrics The University of Florida

2. Agenda Background Introduction Context Methods Data Implications Questions & Discussion

3. Embedded Librarianship Literature Public Services Quarterly special issue 2010 Shumaker & Talley, 2009 – Special Library Association References in the paper Good literature on implementation, not as much on assessment

4. Purpose The purpose of the research project was to: Develop an embedded librarian program to support students in the OnMed program. Evaluate the program using a case study approach. Provide empirical evidence to support the efficacy of embedded librarians programs. Guide future embedded librarian implementations and evaluations.

5. Overview of the Case Institution Program Course Participants

6. Institution UF Fun Facts Enrollment > 50,000 students: including undergraduate, graduate, and professional 16 Colleges and > 150 research centers and institutes 5,434 faculty members $619 million in research awards in 2010-2011 Hundreds of commercial products licensed including Gatorade®

7. Program: OnMed Collaboration between Colleges of Medicine and Education Online graduate degree program for clinical health science educators Goal: improve the incorporation of educational technology into the educational practices of clinical health science educators Program: 36 credits 9 transfer from terminal degree Courses were designed to align with ACGME Competencies

8. Setting: UF Health Science Center

9. 6 Academic Colleges

11. Setting: UF College of Education Key College of Education Facts and Figures Enrollment: 1,863 full-time students at all levels Online Enrollment: 4,395 Degree Programs: 21 Online Degree Programs: 12 degrees offered, all graduate – including 4 online doctoral (EdD) programs Faculty: 106 full-time Research Funding: $37.7 million total

12. Course 8 week online course Moodle Primarily asynchronous 3rd course in the program Critical analysis of academic research was a primary course objective Course assignments included weekly discussions and a brief annotated bibliography

13. Participants N=6 Participants were clinicians and instructors in the Colleges of Medicine, Pharmacy, and Dentistry. College of Medicine Departments: – Pediatrics (N=3) – Ophthalmology (N=1)

14. Implementation Overview

15. Instructor Collaboration Collaboration began when the course was being designed and continued through to the end of the course and beyond. Involvement with the librarian was well documented – in the syllabus and within Moodle. The instructor provided input and recommendations for assessing the librarian’s impact.

16. Moodle Integration

17. WeekCourse ModuleLibrary Content 1Introduction1) Forum Introduction 2) Pre-Assessment 2Reading Research1) RefWorks Handout 2) Refworks Elluminate Session 3) What is an Annotated Bibliography 4) Critical Analysis LibGuide 3Research QuestionsN/A 4Mid-Point1) Peer Review & Ulrich's 2) Searching Ulrich's Video 3) Journal Citation Reports Video 5Building Your Annotated Bibliography 1) ERIC Demonstration Video 2) Why Use MeSH Video 3) MeSH Demonstration Video 6Working on Your Annotated Bibliography Search Support: 1) Presentation on finding medical education literature 2) General database search techniques 7Annotated Bibliography Continued Search Support 8Issues & Current Research Post-Assessment

18. Example

19. Optional Interaction Meebo Chat Widget Discussion Board

20. Research Questions How does the presence of an online embedded librarian influence graduate students' experience in an online educational technology research course as defined by: Self-efficacy with information literacy and library skills Library skill performance Quality of research as measured by citation analysis Reflections on the experience

21. Methodology Mixed Methods Case Study Purpose - Exploratory & Evaluative

23. Data Collection Pre and Post Assessments Citation analysis Learner Reflections – 2 open ended reflection questions Instructor Interview Librarian Field Notes

24. Pre-Post Assessments 13 items measuring self- efficacy with information literacy and library skill performance Monoi and O’Hanlon & Diaz, 2005 Delivered online via Zoomerang Paired data - analyzed using a paired t-test

25. Citation Analysis Objective Rubric Subjective Rubric Tunon & Brydges, 2006; 2009

26. Reflection Questions Students were asked to respond anonymously to these questions: What role did the librarian (who offered instructional support and assistance) play in your process of completing the annotated bibliography assignment? In what ways did your prior experiences in clinical research and literature searching influence your completion of the annotated bibliography assignment and use of the embedded librarian support?

27. Findings - Overview Learners Used the Instructional Materials Positive Impact Demonstrated by: – Increased Self-Efficacy and Performance – High Quality Annotated Bibliographies – Themes from the Qualitative Data

28. Quantitative: Resource Access

29. Quantitative: Pre-Post & Citation Analysis

30. Qualitative Themes 1.The nature of the annotated bibliography assignment and critical analysis required in the annotations 2.Use of the library instructional materials 3.Literature search process: search terms and strategies 4.Process of finding useful and relevant articles

31. Theme 1: Annotated Bibliography "I approached the annotations as though I was going to use them to write a manuscript which helped" "I was really not getting it until I realized that it is essentially a summary of what I grasp from a research paper when I am reading it to quote in a manuscript"

32. Theme 2: Instructional Materials "The videos from the librarian were helpful to understand the annotation concept and how it is used." "I certainly utilized some of the resources provided by the librarian...journal citation reports and searching Ulrich's video"

33. Theme 3: Literature Search Process "my confidence in my ability to conduct sound searches" "I struggled a bit during the search" "I still have an elementary understanding of how to decide on terms and combine them in such a way as to produce a useful list of articles"

34. Theme 4: Challenges Finding Relevant Articles "Finding the correct search terms to get the articles that answered my question" "Challenges I had with this assignment were finding publications that were relevant but not repetitive" "Literature on it [search topic] has literally exploded so it is hard to sort through what is relevant and what is not”

35. Triangulation Positive Impact - – Supported by the pre and post- implementation assessments and learner reflections Learner-Content Interactions - – Supported by Moodle access log data, learner reflections, and instructor interview

36. Additional Findings The importance of faculty-librarian collaboration was another theme that emerged from the data Access to the course syllabus and content was critical to the successful design of the content The instructor supports library-faculty collaborations and make recommendations for future collaborations

37. Implications Professional Practice Administration & Policy Curricula

38. Implications for Librarianship Collaboration with faculty Selecting courses carefully Balancing User Needs & Priorities Educating Future Librarians

39. Policy Implications Maximize Impact to Gain ROI: – Reusable Materials – Templates – Scholarship of Teaching and Learning (SoTL) Accreditation Standards and Information Literacy Supporting Patient Care

41. Curricular Implications for the HSC Library Evidence based medicine Importance of measuring long term impact

42. Directions for Future Research Possible Methodologies for Future Studies: – Case Studies of Other Contexts – Experimental and Quasi-Experimental Designs to control for other factors

43. Directions for Future Research Potential Research Questions: – In what stages of the research process could librarians benefit from interacting with a librarian? – What could the librarian have done differently, especially to increase learner-instructor interactions?

44. Conclusions Online embedded Librarians are the Future! Embedded Librarian Implementations: – Should Include customized instructional content and opportunities for learner-content and learner- instructor “librarian” interactions – Are context dependent – Evaluation is essential