Searching for studies in complex areas Mary Dixon-Woods Department of Health Sciences University of Leicester Andrew Booth, University of Sheffield

A typical definition of an SR "A systematic review is a review in which there is a comprehensive search for relevant studies on a specific topic, and those identified are then appraised and synthesised according to a pre-determined explicit method Klassen, Jadad, & Moher, 1998

Technical problems of searching Effort invested in indexing systems and filters on electronic databases for quantitative research Qualitative research still very elusive; abstracting is a problem (Shaw et al, 2004) But searching isnt just a technical difficulty

Greenhalgh, 2005 Protocol driven: 150 (30) Electronic database search*126 (25) Hand search (32 journals) 24 (5) "Snowballing": 252 (51) Reference tracking 218 (44) Citation tracking 34 (7) Personal knowledge: 119 (24) Sources known to research team 85 (17) Social networks of research team 29 (6) Serendipitous 5 (1) Total in final report 495

CIS approach Fixed, protocol-driven approach to searching has advantages and limitations Emphasises more creative, organic, intuitive forms of searching Accepts that exhaustiveness might not be possible or appropriate Recognises that it may not always be possible to be transparent

Treat the question as a compass rather than an anchor –Eakin & Mykhalovskiy (2003) Not Big Bang searching Iterative Spiral Searching Subjecting a question to continual review and refinement

This requires: Purposively searching for new insights (new databases, new disciplines, new search methods) – active sampling of the literature Lateral leaps rather than more of the same Actively seeking the disconfirming case rather than the averaging effect Citation searching within divergent disciplines/schools of thought Challenge: –Sampling research is fundamentally not like sampling people. Unlike people, research papers have a vested interest in being different from one another, and are (in theory at least) only published if they are saying something new.. Dixon-Woods, Bonas, Booth et al, 2006

Searching not reproducible, not necessarily even transparent More we learn about SR methods the more we recognise that reproducible searching is a myth Different questions, search strategies, selection of abstracts, inclusion criteria, selection of articles, methods of synthesis, level of analysis, implications…..and on, and on, and on! Replace with criterion of reproducible with rigorous Reporting of methods is not for reproducibility but to increase our trust of the research

Assessing comprehensiveness The main purpose of a CIS is interpretative not aggregative Sensitivity to evolving question/problem definition Therefore the point of termination is not when you have identified ALL studies It is when you no longer identify NEW insights (analogous to theoretical saturation) Requires iterative, sampling-based approach

A different view of systematic Systematic is, by definition, a system, plan, or organized method (Oxford English Dictionary [OED]), methodical in procedure or plan (Merriam-Webster). If a systematic review is then….about method itself, then qualitative methods include valid principles and procedures with which to progress in reviewing literature systematically. [Jones, 2004] A wide range of qualitative methodical approaches are, therefore, available to perform or carry out a review in a particular field of interestsystematically and meticulously (OED). [Jones, 2004]

Systematic and Explicit Simply means that we follow an (appropriate, but not specific) system and that we communicate what we have done These are not markers of a systematic review – they are characteristics of ALL well-conducted, well-reported research

References - 1 Eakin, J & Mykhalovskiy, E (2003). Reframing the evaluation of qualitative research: Reflections on a review of assessment guidelines in the health sciences. Journal of Evaluation in Clinical Practice, 9(2), Greenhalgh TGreenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ Nov 5;331(7524): Epub 2005 Oct 17.Peacock R Klassen TP, Jadad AR, Moher D. Guides for reading and interpreting systematic reviews. I. Getting started. Arch Pediatr Adolesc Med 1998;152: Arch Pediatr Adolesc Med 1998;152:

References - 2 Booth A (In press) Brimful of STARLITE: Towards standards for literature searches. JMLA (October) Dixon-Woods M, Bonas S, Booth A et al. How can systematic reviews incorporate qualitative research? A critical perspective. Qualitative Research, 6 (1), (2006) Dixon-Woods M, Booth A, Jones D, et al. How can systematic reviews incorporate qualitative research? ESRC Methods Briefing No. 6 pdf pdf Jones K. Mission Drift In Qualitative Research, Or Moving Toward A Systematic Review Of Qualitative Studies, Moving Back To A More Systematic Narrative Review. The Qualitative report, 9(1), Qualitative report91http:// Shaw RL, Booth A, Sutton AJ, Miller T, Smith JA, Young B, Jones DR, Dixon-Woods M (2004). Finding qualitative research: an evaluation of search strategies. BMC Research Methodology 4:5 (16 Mar).Finding qualitative research: an evaluation of search strategies