Project Transform Julian Elliott Australasian Cochrane Centre on behalf of Project Transform

Starting point Content production is our core business Global network of contributors is our key asset Technology alone cannot solve our challenges It is only through the combination of our networks and technology that our long term value and sustainability can be ensured

Goal To significantly improve the long-term value and sustainability of Cochrane by piloting, refining and scaling-up innovations in content production. We expect this to deliver significant benefits for Cochrane review production and updating, improve impact and topic coverage, and enhance the engagement, retention and experience of authors.

Objectives To support Strategy to 2020 and build the infrastructure within Cochrane to deliver high value content. Specific objectives: 1. To more effectively harness the skills and enthusiasm of our contributors (Strategy to 2020, Goal 4) 2. To more efficiently identify research and understand review currency (Strategy to 2020, Goal 1)

Challenges Information retrieval Opportunities for new contributors Connecting contributors Ensuring quality content

Information retrieval The challenge Virtually impossible to find studies for inclusion in a timely and reliable way Consumes a significant proportion of resources invested in review production Significant delays and unnecessary duplication No complete repository of randomized trials

Evidence Pipeline

Getting involved The challenge Pathways for new contributors are poorly defined Poor experience for potential new contributors Time, effort and frustration for Managing Editors No significant increase in the number of new contributors per year

The screening tool You are not alone! Progress bar Three choices (and you can’t go back) We do this using a citation screening tool. This tool is fundamental to the crowd’s ability to perform the task. We wanted to develop something that focused almost entirely on the task in hand – that of screening a citation – as you can see the screen is mostly taken up with the citation which is stripped down to just title and abstract. There are some built in pre-defined highlighted words and phrases which are to help guide screeners to the most relevant parts of a citation. Yellow highlighted words and phrases indicate that the record is likely to be describing an RCT and Red highlights indicate that the record is likely to be a Reject. Screeners can also add their own highlights. There are three decision buttons: RCT/CCT, Reject or Unsure and screeners have to make a decision on a record; two other features I just want to quickly point out are the all important progress bar, and the feature which tells you how many others are online screening citations at the same time… Yellow highlights to indicate a likely RCT Red highlights

The Embase project: recruitment We have a task, we have a tool, we just need a crowd. We’ve not found this difficult. In a year since going live we’ve had over 900 people sign up to take part. The crowd have screened over 110,000 citations and identified 4,000 reports of RCTs. - 900+ people have signed-up to screen citations in 12 months - 110,000+ citations have been collectively screened 4,000 RCTs/q-RCTs identified by the crowd 99% sensitivity and 99% specificity

Connecting contributors The challenge Many author teams have insufficient skills or resources to complete the required range of tasks on time without external assistance Large burden on CRGs to fill this gap Cochrane’s key competitive advantage – our global network of contributors – is not well utilized in this model

Ensuring quality content The challenge We operate in an increasingly competitive environment Increasing recognition that our reviews are of variable quality and many are out of date CRGs work hard to ensure quality and currency, but the volunteer model limits the ability to ensure delivery Cochrane has limited ability to ensure delivery of an up to date, quality product limiting our ability to develop new revenue sources

Inputs Existing variation in content production New opportunities presented by Project Transform and other Cochrane projects Evolving needs of our end users Need to develop new business models

Production Models Review and evaluate various production models in use currently Select one or more for piloting and evaluation Refine and promote across Cochrane Aim: author team arrangements that Build on Cochrane’s strengths Scalable and cost-effective Improves accountability Maintain financial viability

Overview Information retrieval  Evidence Pipeline Opportunities for new  Getting Involved contributors Connecting contributors  Task Exchange Ensuring quality content  Production Models

Complex intervention looking at psychosocial outcomes in dementia Author team characteristics N=4 // Skilled (methods and statistics) // Domain knowledge//10 hrs per week spent on review Current model Project TRANSFORM Multi-source complex search strategies developed based on P and I 90% of search and screen activity already performed 10,000+ search results retrieved 64 already annotated results delivered 1 hr spent screening results 250 hrs screening (approx 6 months) 25 full texts obtained and assessing 12 full texts obtained and assessed 8 included studies 8 included studies Time elapsed from protocol publication: approx. 8 months Time elapsed from protocol publication: approx. 2 weeks

Team Project Executive Julian Elliott (Co-Lead), James Thomas (Co-Lead), Sally Green, Chris Mavergames, Steve McDonald, Anna Noel-Storr, David Tovey Project Team Clive Adams, Lorne Becker, Linn Brandt, Rachel Churchill, Agustin Ciapponi, Miranda Cumpston, Gordon Dooley, Ruth Foxlee, Demian Glujovsky, Toby Lasserson, Geraldine Macdonald, Sue Marcus, Rupert McShane, Charlotte Pestridge, Daniel Perez Rada, Gabriel Rada, Jessica Thomas and IKMD developers Project component leads Evidence Pipeline: James Thomas, Steve McDonald Getting Involved: Anna Noel-Storr, Chris Mavergames Task Exchange: Anna Noel-Storr, Chris Mavergames, Julian Elliott Cochrane Evidence Networks: Julian Elliott, David Tovey

Participate It is only with the involvement of the broader Cochrane community that we will be able to realise the opportunities presented by this work. Information Regular updates in the Cochrane Community and Cochrane Connect newsletters A website and email group which will be kept up to date as the project progresses Regular webinars  Workshops and presentations at major meetings Get involved Interaction with Cochrane groups, including executives, boards, review groups Opportunities to get involved in early testing and piloting  Opportunities for computer scientists to get involved in the machine learning aspects of the project

Project Transform contacts www.cochrane.org/transform transform@cochrane.org