1. Alison O'Mara-Eves EPPI-centre, UCL Institute of Education
Managing the ‘information deluge’: How text mining and machine learning are changing systematic review methods
Alison O'Mara-Eves
EPPI-centre, UCL Institute of Education

2. Acknowledgements & declaration of interest
Many people to acknowledge, especially James Thomas & Claire Stansfield, but also Ian Shemilt, Sergio Graziosi, Jeff Brunton (all from EPPI-Centre)
Parts of our team’s work on text mining and machine learning for systematic reviews are (or have been) funded by:
Medical Research Council (UK),
Cochrane Collaboration,
JISC,
National Health & Medical Research Council (Australia),
Wellcome Trust,
Bill & Melinda Gates Foundation.
All views expressed are my own, and not necessarily those of these funders.

3. Data deluge: Increasing amount of research
More information than policymakers and practitioners have time to digest

4. 1972
1976

5. Systematic review
Avoid bias
Rigorous
Transparent
Replicable

6. Systematic review process
Search for all possibly relevant studies (or best sample)
‘Screen’ studies for inclusion in the review
Extract information from the studies (e.g., about the population, intervention, outcomes, research methods used)
Analyse (synthesise) the evidence
Draw conclusions across the body of evidence

7. Over time, however, things have changed: exponentially increasing amount of research, more accessible

8. The data deluge grew and now
systematic reviews are taking too long
And costing too much £££

9. Meanwhile…
Developments in computer science technologies: text mining, natural language processing, machine learning, artificial intelligence

10. Exploring whether technology can help with different review tasks, including:
Searching
Screening
Extracting information
Analysis

11. ** Searching Sample of citations Citation elements
(title, abstract, controlled vocabulary, body of text, etc)
Text analysis
Word frequency counts, phrases or nearby terms in text
Generic tools
Database specific (PubMed) tools
Term extraction and automatic clustering
Statistical
analysis
TF-IDF
Statistical and linguistic analysis
TerMine
Automatic Clustering
Some worked examples of using this for complex search strategies for a recent,
Bibliography/ tools AHRQ, Res Synth Methods, HLWIKI
Word or phrase lists
Visualisation
Revise search elements
Humans assess relevance and impact to search

12. ** Screening Has received most R&D attention
Diverse evidence base; difficult to compare evaluations
‘semi-automated’ approaches are the most common
Possible reductions in workload in excess of 30% (and up to 97%)
Summary of conclusions
Screening prioritisation
‘safe to use’
Machine as a ‘second screener’
Use with care
Automatic study exclusion
Highly promising in many areas, but performance varies significantly depending on the domain of literature being screened

13. Does it work? e.g. reviews from Cochrane Heart Group

15. Pre-built or build your own
Developed from established datasets
RCT model
Systematic review model
Economic evaluation
Build your own

16. Pre-built classifier
An RCT classifier was built using more than 280,000 records from Cochrane Crowd
In EPPI-Reviewer 4 software
60% of the studies have scores < 0.1
If we trust the machine, and automatically exclude these citations, we’re left with % of the RCTs (i.e. we lose 0.1%)

17. ** Extracting information (example)
Characteristics of studies:
Population
Intervention
Outcomes
Graphic indicates the extent of similarity of studies
Presents the snippets of text on which the judgements are based

18. ** Analysis (example)

19. Cochrane Collaboration: an example of an ‘early adopter’
How are text mining and machine learning changing systematic review methods?
Cochrane Collaboration: an example of an ‘early adopter’
“Animated Storyboard: What Are Systematic Reviews?". cccrg.cochrane.org. Cochrane Consumers and Communication.

21. A PICO ‘ontology’ is being developed in Cochrane
… and is being applied to…

22. … all Cochrane reviews and all the trials they contain

23. … Boolean searches are replaced by the specification of the ‘PICO’ of interest

24. PICOFINDER

25. Through a combination of human and machine effort the aim is to identify and classify ALL trials using this system.
Identifying studies for systematic reviews (of RCTs) will then be a simple process of specifying the relevant PICO
Very challenging to automate

26. Cochrane Register of Studies: Triaging of relevant studies to different Cochrane Review Groups
CRS-Web

27. How are text mining and machine learning changing systematic review methods?
Other examples of early adopters

28. Updating reviews/ clinical guidelines
Currently working with NICE to set up a ‘surveillance’ system
Aims to identify new evidence as soon as it’s published and automatically identify which guideline it relates to
Eventually, hope to automatically extract information – including results – so that we’re able to say how likely it is that a given new study will change the evidence and suggest that the guideline needs updating

29. Behavioural science

30. In summary: How are text mining and machine learning changing systematic review methods?
From ‘search strategy’ to PICO definition?
From ‘data extraction’ to structured data?
Surveillance systems for updating reviews and guidelines
Synthesise evidence in ‘real time’
The ‘systematic review’ will become a matter of ascertaining the validity and utility of combining particular sets of studies at particular points in time, rather than the tedious trawling for, and extraction of, data

31. Key considerations Are there risks, and are we willing to take them?
Could using technologies lead to
Introduction of bias
Loss of comprehensiveness
Reduction in transparency?
Reviews are more timely and less resource-intensive

32. Selected bibliography
SR Toolbox
Paynter R, et al. (2016). EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews. AHRQ Research White Paper.
O'Mara-Eves A, et al. (2015). Using text mining for study identification in systematic reviews: a systematic review of current approaches. Syst Rev 4: 5.
Thomas J, et al. (2011). Applications of text mining within systematic reviews. Res Synth Meth 2(1): 1-14.
Shemilt I, et al. (2016) Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Syst Rev 5: 140.
Stansfield C, et al. (2017). Text mining for search term development in systematic reviewing: a discussion of some methods and challenges. Res Synth Meth., DOI: /jrsm.1250
Stansfield C, et al. (2015) Reducing systematic review workload using text mining: opportunities and pitfalls. J. EAHIL 11(3): 8-10.

33. Alison O'Mara-Eves EPPI-Centre UCL Institute of Education