1. Reproducibility of Search Strategies in Systematic Reviews Preliminary Results Jonathan Koffel, MSI Bio-Medical Library, University of Minnesota Melissa Rethlefsen, MLIS AHIP Eccles Health Sciences Library, University of Utah

2. Rise of the Systematic Review Reporting Guidelines: PRISMA MOOSE Cochrane Institute of Medicine

3. Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched. Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated. Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 2009

4. “The Scopus, EBSCOhost, Ovid, and Web of Science platforms were searched.” Pediatrics. 2012 Mar;129(3):548-56

5. Previous Research Good reporting of some search elements 2 –Database names, search terms Poor reporting of other elements 2 –Date search executed, database platform Reproducibility is heterogeneously defined, but generally low 3,4 2 Sampson et al. J Clin Epidemiol. 2008 Aug;61(8):748-54. 3 Yoshii et al. J Med Libr Assoc. 2009 Jan;97(1):21-9. 4 Golder et al. J Clin Epidemiol. 2013 Mar;66(3):253-60

6. Purpose 1.Examine search reporting and reproducibility in systematic reviews published in high-impact medicine journals in 2012 2.Investigate the impact of librarian involvement and reporting guidelines on reporting/reproducibility rates

7. Hypotheses 1.Reporting/reproducibility rates will be higher when a librarian was involved. 2.Reporting/reproducibility rates will be higher when a guideline was mentioned or required by the journal.

8. Journals Included Cardiology Circ Cardiovasc Genet Circ Cardiovasc Interv Circ Heart Fail Circ Res Circulation Eur Heart J J Am Coll Cardiol JACC Cardiovasc Imaging JACC Cardiovasc Interv Nat Rev Cardiol Surgery Am J Surg Pathol Am J Transplant Ann Surg Ann Surg Oncol Arch Surg Br J Surg Endoscopy J Am Coll Surg J Neurol Neurosurg Psychiatry Surg Obes Relat Dis Pediatrics Arch Dis Child Arch Dis Child Fetal Neonatal Ed Arch Pediatr Adolesc Med Eur Child Adolesc Psychiatry J Am Acad Child Adolesc Psychiatry J Pediatr Pediatr Allergy Immunol Pediatr Infect Dis J Pediatrics Semin Fetal Neonatal Med

9. Identification of Studies PubMed search on 9/2/2013 using modified Montori strategy 5 (search*[Title/Abstract] OR meta-analysis[Publication Type] OR MEDLINE[Title/Abstract] OR EMBASE[Title/Abstract] OR Meta- analysis[Title/Abstract] OR (systematic[Title/Abstract] AND review[Title/Abstract]) AND (("2012/01/01"[PDat] : "2012/12/31"[PDat]) PubMed SR subset filter also used for comparison –<4% of included articles found only by Montori strategy 5 Montori et al. BMJ. 2005 Jan 8;330(7482):68.

10. Identification of Studies All retrieved studies screened by both authors to identify systematic reviews –Searches at least one published literature database –Inclusion/exclusion criteria are prespecified –Does not limit to a certain number of journals or subset of journals Disagreements resolved by review of original articles and consensus

11. CardiologySurgeryPediatrics 216 317201 51 14277 Retrieved Included 270

12. Items Identified Items covered: 1.Search elements (eg., terms, strategy, limits) 2.Librarian involvement (author and role) 3.Guideline mentioned

13. Data Extraction Data extraction form created in Qualtrics Included articles split between authors Initial entry completed in March, 2014

14. Analyses SPSS used for all analyses –Logistic regression (odd ratio) –Chi Square/Fisher (p-value) –Mann-Whitney/T-Test (means) –α =.01

15. No significant differences based on discipline

16. * * * * * * p<.01 for comparison Sampson et al. J Clin Epidemiol. 2008 Aug;61(8):748-54.

17. Librarian Involvement P<.001P<.01 P<.001

19. Conservative Reproducibility Includes full search strategy for at least one database –Database searched –Provider –Date range –Search date –Terms –Booleans –Indication of limits

20. Liberal Reproducibility Includes core search elements –Database –Date range –Specific terms –Indication of limits

21. Reproducibility Rates Reproducibility rates <8% in prior studies –Previous criteria mirrored conservative criteria ConservativeLiberal Surgery11%56% Pediatrics12%57% Cardiology12%39% Total11%53%

22. LibrarianGuidelineJournal Database Namedns Databases SearchedMD 1.34 (.78-1.89)***ns Provider Namedns Search Years Indicatedns Search Datens Search Updatedns Search TermsOR.2 (.08-.49)***ns Full Strategy Presentedns Booleanns Limitsns Conservative Reproducibility ns Liberal Reproducibilityns **p<.01 ***p<.001 Discipline included in regression

23. Librarian Author/NoneLibrarian Text/None Database Namedns Databases SearchedMD 2.8 (1.8-3.9)***MD 1.3 (.7-2.0)*** Provider Namedn/ans Search Years Indicatedns Search Datens Search UpdatedOR.08 (.01-.52)**ns Search TermsOR.16 (.04-.71)**OR.18 (.07-.48)*** Full Strategy Presentedns Booleanns Limitsns Conservative Reproducibility ns Liberal Reproducibilityns **p<.01 ***p<.001 Discipline, Journal Requirements and Guideline Mentioned included in regression

24. Results Reproducibility is not predicted by: –Librarian involvement –Journal requirements –Guidelines mentioned in the paper

25. Results Librarian Involvement –Increases number of databases searched –Decreases updating of search results when librarian is an author –Decreases reporting of search terms

26. Limitations Published information –Librarian could be involved, but not mentioned Ambiguous search elements –Booleans? Limits? Full strategy? Limited to three medical disciplines and a single year

27. Next Steps Duplicate entry of data Examination of reporting in greater depth –Publication lag –Presentation of elements for more than one database

28. Conclusions 1.Librarian involvement does not intrinsically mean better reporting 2.We must work with authors and publishers to improve reporting

29. Thank You Questions? Jonathan Koffel jbkoffel@umn.edu