1. Enhancing Students’ Literature Searches by Integrating Personal Librarian Feedback into a PBL Course Suzanne Edmunds, MD, C. Randall Clinch, DO, MS, David Stewart, MSLS, Patrick Reynolds, MD

2. Wake Forest School of Medicine Background Efficient access and application of information from electronic resources are essential skills for medical professionals as lifelong learners Problem based learning (PBL) courses provide an ideal environment for developing the skills involved in information retrieval and management (Boumarafi, Med Ref Serv Q 2010; Mi, Med Ref Serv Q 2011)

3. Background Our medical school has a personal librarian program that has been in existence since 2008, but had been underutilized up to this point We have 10 personal librarians on staff

4. Wake Forest School of Medicine Background Our objective was to determine the impact of personal librarian feedback during a first-year PBL course on student performance on: search strategies citation accuracy Our study was conducted between November 2010 and February 2011

5. Wake Forest School of Medicine Methods A Literature Search Worksheet (LSW) was developed as a tool for communication between students and the personal librarians Students developed learning issues during a PBL case presented on a Monday Students utilized the LSW to describe their: search strategy and results the amount of time spent on their search to provide citations

6. Wake Forest School of Medicine Methods The LSW outlines how search was conducted: Which search engines were used? Limits? Keywords? Results? Time spent? Two complete citations Were learning issues answered completely?

7. Wake Forest School of Medicine Methods Student completes LSW & submits to PL Librarian reviews LSW LSW is returned to student electronically

8. Methods Librarians evaluated search Logical progression of steps Appropriateness of search engines employed Citations Accuracy Quality, relevance Librarian training sessions performed prior to scoring LSWs Scores kept in secure database Free text comments provided to students as feedback

9. Wake Forest School of Medicine Methods - analysis 122 students in the Class of 2014 participated in 6 exercises between 11/10 – 2/11 Variables analyzed: PL scoring of search question (1-5 scale; Poor-Excellent) PL scoring of search strategy (1-5 scale; Poor-Excellent) Limits used during search (Y/N) Time spent on learning issue task ( 120min)

10. Wake Forest School of Medicine Methods – analysis (cont’d) Variables analyzed: (cont’d) Student rating of search success (1-3 scale; Unsuccessful, Partially successful, Successful) PL rating of search success (1-3 scale; Unsuccessful, Partially successful, Successful) PL rating of completeness of citations submitted (Y/N) Comparison of Class of 2014 vs Class of 2013 on same variables listed above Each class had same learning issue on a standardized patient examination

11. Results – Class of 2014 VariableResponse optionsP-value Quality of Search Question1-5 (Poor - Excellent)0.001* Quality of Search Strategy1-5 (Poor - Excellent)<0.001** Limits used on first stepYes/No0.117 £ Complete citationsYes/No<0.001 £ Student Rating of Search Success1-3 (Unsuccessful - Successful)<0.001** Librarian Rating of Search Success1-3 (Unsuccessful - Successful)<0.001* Time Spent on Learning Issue1-3 ( 120min)<0.001** *Repeated Measures ANOVA (GreenhouseGeisser correction) **Repeated Measures ANOVA £ McNemar test

17. Completeness of Citations Exercise #1Exercise #6

18. Results – Class of 2013 (n=114) vs. 2014 (n-122) VariableResponse optionsP-value* Quality of Search Question1-5 (Poor - Excellent)0.219 Quality of Search Strategy1-5 (Poor - Excellent)0.047 Limits used on first stepYes/No0.867 Complete citationsYes/No0.668 Student Rating of Search Success1-3 (Unsuccessful - Successful)0.144 Librarian Rating of Search Success1-3 (Unsuccessful - Successful)0.196 Time Spent on Learning Issue1-5 ( 120min)<0.001 *Pearson Chi-Square

19. Time Spent Class of 2013Class of 2014

20. Wake Forest School of Medicine Results n=122; 60% response rate

21. Wake Forest School of Medicine Results n=122; 60% response rate

22. Wake Forest School of Medicine Results n=122; 60% response rate

23. Wake Forest School of Medicine Results n=122; 60% response rate

24. Wake Forest School of Medicine Results Librarian Feedback 8/10 librarians thought LSW was worthwhile teaching tool Most felt 3 or 4 LSW’s appropriate 8/10 looked forward to continued participation in PBL course

25. Wake Forest School of Medicine Discussion Improvement in quality of search questions, quality of search strategies, efficiency of searches, and student and librarian rating of search success all noted over sessions 2-5 (Class of 2014) Improvement in citation accuracy at completion of study (Class of 2014) Librarians rated search strategies of Class of 2014 higher than Class of 2013 Class of 2014 spent longer on searches than Class of 2013

26. Wake Forest School of Medicine Discussion Was grading more lenient on first session? Did students and librarians suffer from overexposure fatigue with sixth session? Difficult to interpret “time spent” as marker of efficiency

27. Wake Forest School of Medicine New Directions Three LSW’s over four months Librarians are assigned to PBL group rather than individual After reviewing LSW’s, librarians attend student- driven session with personalized feedback for that group Will continue to re-evaluate program and revise as needed