Alison O'Mara-Eves EPPI-centre, UCL Institute of Education Managing the ‘information deluge’: How text mining and machine learning are changing systematic review methods Alison O'Mara-Eves EPPI-centre, UCL Institute of Education

Acknowledgements & declaration of interest Many people to acknowledge, especially James Thomas & Claire Stansfield, but also Ian Shemilt, Sergio Graziosi, Jeff Brunton (all from EPPI-Centre) Parts of our team’s work on text mining and machine learning for systematic reviews are (or have been) funded by: Medical Research Council (UK), Cochrane Collaboration, JISC, National Health & Medical Research Council (Australia), Wellcome Trust, Bill & Melinda Gates Foundation. All views expressed are my own, and not necessarily those of these funders.

Data deluge: Increasing amount of research More information than policymakers and practitioners have time to digest

1972 1976

Systematic review Avoid bias Rigorous Transparent Replicable

Systematic review process Search for all possibly relevant studies (or best sample) ‘Screen’ studies for inclusion in the review Extract information from the studies (e.g., about the population, intervention, outcomes, research methods used) Analyse (synthesise) the evidence Draw conclusions across the body of evidence

Over time, however, things have changed: exponentially increasing amount of research, more accessible

The data deluge grew and now systematic reviews are taking too long And costing too much £££

Meanwhile… Developments in computer science technologies: text mining, natural language processing, machine learning, artificial intelligence

Exploring whether technology can help with different review tasks, including: Searching Screening Extracting information Analysis

** Searching Sample of citations Citation elements (title, abstract, controlled vocabulary, body of text, etc) Text analysis Word frequency counts, phrases or nearby terms in text Generic tools Database specific (PubMed) tools Term extraction and automatic clustering Statistical analysis TF-IDF Statistical and linguistic analysis TerMine Automatic Clustering Some worked examples of using this for complex search strategies for a recent, Bibliography/ tools AHRQ, Res Synth Methods, HLWIKI Word or phrase lists Visualisation Revise search elements Humans assess relevance and impact to search

** Screening Has received most R&D attention Diverse evidence base; difficult to compare evaluations ‘semi-automated’ approaches are the most common Possible reductions in workload in excess of 30% (and up to 97%) Summary of conclusions Screening prioritisation ‘safe to use’ Machine as a ‘second screener’ Use with care Automatic study exclusion Highly promising in many areas, but performance varies significantly depending on the domain of literature being screened

Does it work? e.g. reviews from Cochrane Heart Group

Pre-built or build your own Developed from established datasets RCT model Systematic review model Economic evaluation Build your own

Pre-built classifier An RCT classifier was built using more than 280,000 records from Cochrane Crowd In EPPI-Reviewer 4 software 60% of the studies have scores < 0.1 If we trust the machine, and automatically exclude these citations, we’re left with 99.897% of the RCTs (i.e. we lose 0.1%)

** Extracting information (example) Characteristics of studies: Population Intervention Outcomes Graphic indicates the extent of similarity of studies Presents the snippets of text on which the judgements are based

** Analysis (example)

Cochrane Collaboration: an example of an ‘early adopter’ How are text mining and machine learning changing systematic review methods? Cochrane Collaboration: an example of an ‘early adopter’ “Animated Storyboard: What Are Systematic Reviews?". cccrg.cochrane.org. Cochrane Consumers and Communication.

A PICO ‘ontology’ is being developed in Cochrane … and is being applied to…

… all Cochrane reviews and all the trials they contain

… Boolean searches are replaced by the specification of the ‘PICO’ of interest

PICOFINDER https://youtu.be/WtqAnL6QPt4

Through a combination of human and machine effort the aim is to identify and classify ALL trials using this system. Identifying studies for systematic reviews (of RCTs) will then be a simple process of specifying the relevant PICO Very challenging to automate

Cochrane Register of Studies: Triaging of relevant studies to different Cochrane Review Groups CRS-Web

How are text mining and machine learning changing systematic review methods? Other examples of early adopters

Updating reviews/ clinical guidelines Currently working with NICE to set up a ‘surveillance’ system Aims to identify new evidence as soon as it’s published and automatically identify which guideline it relates to Eventually, hope to automatically extract information – including results – so that we’re able to say how likely it is that a given new study will change the evidence and suggest that the guideline needs updating

Behavioural science

In summary: How are text mining and machine learning changing systematic review methods? From ‘search strategy’ to PICO definition? From ‘data extraction’ to structured data? Surveillance systems for updating reviews and guidelines Synthesise evidence in ‘real time’ The ‘systematic review’ will become a matter of ascertaining the validity and utility of combining particular sets of studies at particular points in time, rather than the tedious trawling for, and extraction of, data

Key considerations Are there risks, and are we willing to take them? Could using technologies lead to Introduction of bias Loss of comprehensiveness Reduction in transparency? Reviews are more timely and less resource-intensive

Selected bibliography SR Toolbox http://systematicreviewtools.com/ Paynter R, et al. (2016). EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews. AHRQ Research White Paper. O'Mara-Eves A, et al. (2015). Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev 4: 5. Thomas J, et al. (2011). Applications of text mining within systematic reviews. Res Synth Meth 2(1): 1-14. Shemilt I, et al. (2016) Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Syst Rev 5: 140. Stansfield C, et al. (2017). Text mining for search term development in systematic reviewing: a discussion of some methods and challenges. Res Synth Meth., DOI: 10.1002/jrsm.1250 Stansfield C, et al. (2015) Reducing systematic review workload using text mining: opportunities and pitfalls. J. EAHIL 11(3): 8-10.

Alison O'Mara-Eves EPPI-Centre UCL Institute of Education a.omara-eves@ucl.ac.uk