Electronic literature searching for qualitative research Rachel L Shaw with Andrew Booth, Alex J Sutton, Mary Dixon-Woods, David R Jones, Tina Miller, Jonathan A Smith & Bridget Young

The Project How can systematic reviews incorporate qualitative research? 1)Searching the literature 2)Appraising the evidence 3)Synthesis of qualitative evidence 4)Synthesis of qualitative & quantitative evidence –(Dec 02-Oct 04)

Aims of Phase 1 To systematically search the literature for qualitative evidence in order to update an existing Cochrane Review of quantitative evidence on support for breast- feeding women To evaluate the recall & precision of 3 electronic search strategies using 6 bibliographic databases for identifying qualitative research about breast- feeding support

Search strategies 1)Subject headings (thesaurus): using 20 thesaurus terms specific to each database 2)Free text terms: using over 40 qualitative methodology terms & authors 3)Broad-based: using 3 generic terms, qualitative, interviews, findings

Bibliographic databases Medicine: –MEDLINE –EMBASE Nursing: –CINAHL –British Nursing Index Social Sciences: –ASSIA –Social Sciences Citation Index

Definitions Recall: –Potentially relevant studies (tested positive) –How well a search strategy identifies what is relevant to the research question Precision: –Actually relevant studies (diagnosed positive) –How well a search strategy screens out what is not relevant

Discard duplicates Discard non-human Total initial yield N=7420 Raw yield N=17045

Abstract screening Records were screened against 2 criteria: –Topic relevance: breast-feeding support –Methodology relevance: qualitative methods (studies using mixed methods – qualitative & quantitative – included at this stage)

Discard duplicates Discard non-human Total initial yield N=7420 Raw yield N=17045 Abstract screening Screened positive sample N=587

Getting strict on method Many mixed methods studies identified during abstract screening (predominantly quantitative with small qualitative component) Decision taken to limit inclusion to qualitative studies only (& if mixed methods used then qualified only if predominantly qualitative) This demonstrates the iterative process required in such a new field of literature searching

Defining qualitative research Qualitative research is an umbrella term for diverse research methods in many different disciplines An operational definition? We defined qualitative research by what it is NOT: e.g. research that is dependent on numeric analysis was screened not relevant

Discard duplicates Discard non-human Total initial yield N=7420 Raw yield N=17045 Abstract screening Screened positive sample N=587 Strict on method Candidate sample N=262

Calculating recall & precision Modelled from the screening test formula Recall ~ sensitivity –Potentially relevant (tested positive) records from each strategy as a percentage of initial total yield Precision ~ positive predictive value –Actually relevant (diagnosed positive) records as a percentage of potentially relevant records (tested positive)

Potentially relevant (tested positive) records identified by each strategy Recall: potentially relevant records as % of total initial yield 1) Subject headings % (3537/7420) 2) Free-text % (3451/7420) 3) Broad- based % (3912/7420) Total initial yield 7420-

Actually relevant records retrieved (diagnosed positive) identified by each strategy Precision: actually relevant records as % of potentially relevant records 1) Subject- headings % (191/3537) 2) Free-text1724.9% (172/3451) 3) Broad-based1874.7% (187/3912) Total actually relevant records 262-

Raw yield N=17045 Total initial yield N=7420 Broad-based strategy has highest recall Candidate sample N=262 Subject-headings has highest precision (all strategies have generally poor precision) Many non-human & non-qualitative records identified

Trading relevance against comprehensiveness? Price for high recall is poor precision Trade-off is traditionally expected in information retrieval circles Strategies designed to maximise recall 96% of total initial yield were irrelevant Screening was time consuming & costly

Conclusions Subject headings have recognised high precision – BUT in this case - only slightly higher than broad- based strategy No one strategy was sufficiently comprehensive to identify all relevant records A combination of search terms (subject headings & free-text) is necessary to maximise recall BUT high recall leads to laborious screening for very small return Action needed to increase precision

Recommendations for high recall, high precision strategy Better indexing of qualitative research on bibliographic databases Authors use methodology descriptors to describe their work & use structured abstracts where possible Systematic analysis of individual search terms is required Further research in different substantive areas is required

Future questions & outputs Methods question: –Which search terms are most successful at identifying qualitative research? –Papers in progress for JMLA & J Adv Nurs Substantive question: –What does the qualitative evidence about support for breast-feeding women add to the quantitative evidence in the existing Cochrane Review? –A new review of qualitative & quantitative evidence will be our substantive output.

Further dissemination 47 th Scientific Annual meeting of the Society for Social Medicine, Edinburgh, September 2003 Submitted Finding qualitative research to the BMC Research Methodology open access journal Provisionally accepted The problem of quality in qualitative research by Quality & Safety in Health Care In progress: –Using citation searching to identify qualitative research for the International Journal of Social Research Methodology –Defining qualitative research for Qualitative Research