Searching for systematic reviews: a comparison of methods by Diana Papaioannou, Anthea Sutton, Chris Carroll, Ruth Wong & Andrew Booth ScHARR, University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA. Tel: Comparison of Search Methods Discussion Conclusion Traditional systematic review searching i.e. main search Whilst the bulk of the literature in the review was identified via our comprehensive main search strategy, some papers were missed. Papers were missed due to ambiguity of terms & poor indexing. Another problem was the failure of abstracts of “potential includes” to state if the population was employed- meaning they weren’t picked up via our search strategy (which included work-related terms). Citation searching & reference tracking Citation searching of included studies identified by the main search strategy proved an effective way of identifying additional references. Citation searching identified references included in the main search but more importantly identified some unique papers that the main search had failed to find. In addition, citation searching facilitated serendipitous discovery of further papers by unstructured searching. Reference checking proved useful by identifying 4 unique papers to be included in the review. Why Pearl Growing proved problematic: Pearl growing proved problematic in the field of social science as the literature was spread across many multi- disciplinary databases. This limits the use of pearl growing due to difficulties in structuring and managing the search as well as time implications. Pearl growing may perhaps be more suited to areas were terms are well-defined, indexing is consistent and spread of literature is across a limited number of databases. Pearl growing may be more suited to the area of health technology assessment and further research is required to investigate this. Searching in the field of social science requires careful consideration. Papers included in the review were widely spread across a number of multi-disciplinary databases. Problems in indexing, ambiguity of terms and limited abstract content meant the reliability of our main search identifying all potential references was dubious. Follow-up techniques like reference tracking and citation searching were minimally time-intensive but productive, yielding unique reference for inclusion in the systematic review. Pearl growing is perhaps better suited to topic areas where terms are well-defined, indexing is consistent and spread of literature is across a limited number of databases. The UK Higher Education Academy is funding an ongoing project on enhancing the student experience of workplace-based e-learning. This involves a systematic review component and the creation of a best practice framework. Method 1: Traditional Systematic Review Searching A full systematic search was conducted as is typical to this type of project. This entailed developing a search strategy including terms around e-learning (the intervention) and the workplace (the setting). The search was carried out on a number of databases from different disciplines, including education (ERIC, British Education Index), business (Emerald Management Reviews), social sciences (IBSS, Social Sciences Citation Index, ASSIA), health (Medline, PsycInfo, Cinahl), information technology (CSA Computer and information systems abstracts), and library and information science (LISA). This systematic search was supplemented by the reference checking of any included papers. Method 2: Alternative Search Techniques In addition to the systematic searching described above, two alternative approaches were also employed, enabling a comparison of techniques that would inform future practice: Pearl-Growing (PG): took place prior to the systematic searching. This entailed identifying “pearls” – e.g. key papers, from a known literature review on the topic of e-learning in the workplace (1) and checking which databases (as outlined above) those pearls were indexed in, in order to select the most appropriate database in which to conduct the PG, i.e. the database that indexed the highest number of pearls. Citation Searching (CS): took place after the systematic searching and involved conducting a citation search for each of the papers included in the systematic review. The databases used for the CS were Google Scholar, Citation Indexes and Cinahl. Systematic Searching The systematic search yielded a total of 3476 references (after duplicates removed). Of these, only 29 were deemed appropriate to be included in the final review. Pearl-Growing The 13 pearls identified were dispersed across 10 databases. Consequently, pearl-growing was not completed because no database indexed a majority of the pearls; instead, small numbers of pearls were spread over many databases 85 references identified by citation searches 75 remained post de- duplication 10 duplicates removed 14 included titles following title & abstract sift 61 references excluded according to inclusion/exclusion criteria 3 references included in main literature search 11 references ‘unique’ to citation searches 3 references included following full-text sift 8 references excluded following full-text sift and data saturation Citation Searching The citation searching yielded 75 references (after duplicates removed). Of the 14 potentially relevant papers, 11 were uniquely identified by this method of searching. 3 of these 11 papers were subsequently included in the review (see figure 1). Reference Checking 4 additional studies relevant to the review were identified from the reference checking of the included studies. Background and Methods Figure 1: Citation Search Results References: (1) Lain, D. & Aston,J. (2004) Literature Review of Evidence on e-Learning in the Workplace. Brighton: Institute for Employment Studies. SourceNo. of included studies ERIC8 CINAHL6 PSYCINFO7 LISA6 EMERALD1 ASSIA2 MEDLINE6 IBSS2 BRITISH EDUCATION4 WoS1 Table 1: Sources of included studies