1. Theme 5. Cochrane Reviews: updating 9:30 – 10:10 Theme overview and CRG experiences of using tools/frameworks 10:10 – 10:30: Q&A 10:30 – 11:00 Working groups 11:00 – 11:15 Feedback

2. Overview 1.How often should Cochrane Reviews be updated? 2.Which Cochrane Reviews should be a priority for updating? 3.How should updates of Cochrane Reviews, and Cochrane Reviews no longer being updated, be classified in The Cochrane Library?

3. Updating frequency CRGs are not able to update all Cochrane Reviews every two years. The reasoning behind updating decisions is not transparent to readers. The burden of updating is ever increasing Cochrane Reviews may become out of date at different times Updating too soon may introduce bias 80% of 617 respondents moderately or strongly supported a move from the current policy of aiming to update all Cochrane Reviews every two years to a ‘needs-based’ approach

4. Prioritising Cochrane Reviews

6. Classification framework

7. Experiences

8. Criteria area QuestionYes(=1)/No(=0) Strategic importanceIs condition of strategic important to MSK Group or funders or users 0/1 Is intervention of strategic important to MSK Group or funders or users0/1 Patient importance/ impactDoes review measure patient important outcomes? i.e. clinical outcomes 0/1 Does intervention have a high potential impact on quality of life? 0/1 Does intervention potentially significantly lower mortality 0/1 Leading cause of burden of disease in Europe (WHO burden of disease DALY list) 0/1 National Spend High spend on condition (including societal costs)?0/1 High spend on intervention or on currently used alternatives?0/1 Presence on list of 20 drugs which had the greatest number of items dispensed, OR greatest Net ingredient cost0/1 Incidence prevalence High incidence or prevalence?0/1 Recent increase in incidence/prevalence?0/1 Emerging evidence/remaining uncertainty Current review conclusions reflect uncertainty?0/1 Significant RCTs/other eligible trials published since last update?0/1 SRs/HTAs covering same ground NOT published (or about to be published) since last update?0/1 Additional Cochrane criteriaLeading cause of burden of disease in the rest of the world i.e. outside of Europe (WHO burden of disease DALY list) 0/1 Review content out of date 0/1 Review has attracted > 10 citations 0/1 Review in top 10 most accessed MSK reviews 0/1 Review addresses important equity or Global Health Issues 0/1 Author available for update 0/1 Total score:0 Original criteria

9. Question Condition Burden of disease, spend, and incidence/prevalence related to the condition Is condition of strategic importance to CCDAN Group or funders or users Leading cause of burden of disease in Europe (WHO burden of disease DALY list) Leading cause of burden of disease in the rest of the world i.e. outside of Europe (WHO burden of disease DALY list) High spend on condition (including societal costs)? High incidence or prevalence? Recent increase in incidence/prevalence? Evidence Emerging evidence / remaining uncertainty Current review conclusions reflect uncertainty? Significant RCTs/other eligible trials published since last update? Are there new trials addressing a specific call for research in previous version of review SRs/HTAs covering same ground NOT published (or about to be published) since last update? Stakeholder NIHR / DOH / NHS High spend on intervention or on currently used alternatives? Presence on list of 20 drugs which had the greatest number of items dispensed, OR greatest Net ingredient cost Review has contributed to a NICE guidelines Related NICE guideline pending / update due soon Is intervention of strategic importance to funders Has intervention been highlighted in guidelines or elsewhere as requiring more research Patients Does intervention have a high potential impact on quality of life? Is the intervention recommended at all in NICE guidelines Is the intervention a first line treatment according to NICE guidelines? Current treatment options have evidence of effectiveness but are not well tolerated by patients Does the intervention have significant known adverse events (score -1) Intervention discontinued / no longer prescribed (score -1) Cochrane Collaboration CCDAN Review has attracted > 10 citations Review in top 10 most accessed CCDAN reviews Review never cited (score -1) Review rarely accessed (score -1) Could this contribute to an MTM / overview Rest of the World (WHO as proxy) Review addresses important equity or Global Health Issues Is there potential for collaboration with or benefit for WHO or other similar body Does this review directly contribute to Evidence Aid The criteria below are to be considered once a review has been prioritised according to the above and so these criteria do not contribute to the scoring Review Quality Methodology Full risk of bias tool used Summary of findings tables used Methods up to date and written in RevMan 5 headings Is the scope of the review question sensible (should it be split/merged) Does the review address its objectives Patient important changes Are patient important outcomes addressed - if not they must be. Are adverse events dealt with - if not they must be Revised criteria

10. Cochrane Airways Group Prioritisation exercise using the decision flowchart and STATA program

11. Context ~250 reviews 166 reviews over 2 years out of date Don’t have a prioritisation policy and make decisions to update based on author availability Looking to: – make decisions transparent – direct limited resources at most important topics

12. What we did Considered 21 reviews with the highest number of hits and cites for prioritisation TSC ran searches ME/TSC/RA screened titles and abstracts Used prioritisation flow chart and STATA tool Extracted outcome data for the STATA tool where possible even if review currently being updated

13. Flow chart

14. Issues with flow chart Seemed very complicated at first, but was OK to use We created a spreadsheet to reflect the flow of work from the decision tool and record comments on reviews We didn't go searching for abstracts or papers if they weren't available; however we felt that the new studies identified represent the bulk of new research for the review Some updates were already in progress and we consulted authors to try to get the number of hits

15. Innovated table to record decisions

16. Issues with STATA tool Possibly too crude? – need to select a single outcome – extract outcome data often from abstract only. Unresolved issues with continuous data and GIV However, once data had been extracted the STATA tool was easy to use and may find application in some circumstances

17. Reviews that reached stage x in the flowchart Stage 1: 4 review questions appeared answered or partly answered Stage 2: 13 reviews needed some sort of work. 7 were unclear. Most needed SOF Stage 3: We did searches for all of them to test out the STATA tool (so could save time here in future). Some searches challenging to go through STATA: we got sufficient data to run 6 studies through STATA tool but could not analyse the continuous data or the GIV data for unresolved technical reason

18. Outcomes 8 indicated for update for other reasons, 5 possible updates and 8 not to be updated (already up-to-date, irrelevant review question, not enough new data) We didn’t feel that the STATA tool was worth the additional workload and that it was too crude for our needs, however now we know how it works, may use in future

19. Issues/comments Estimate it took around 5 days in total to do this work. Plus 2 hours with a statistician on STATA. This is a reasonable amount of time compared to the editorial process for an ill-conceived update! The flow-chart tool provided – a helpful structure to assess each review – a transparent way to summarise evidence to stakeholders

20. Issues/comments Other groups may need to modify tool and consider which of their reviews to put through the tool We would need a further conversation with an advisory board on the obtained information to rank them/decide relevance of review question Need a transparent way to select reviews for this tool With more time, may be possible to screen references carefully enough to be of use to authors

21. Objectives  To assess whether CRGs other than the CIDG could classify their Cochrane Reviews using the framework  To identify any problems with the framework and potential solutions  To determine how long it takes CRGs to classify Cochrane Reviews Methods  CRGs who nominated themselves classified a sample of 20 Cochrane Reviews selected by CEU, and fed back experiences

22. Results  Thirteen CRGs completed the pilot  CRGs were able to classify Cochrane Reviews of interventions, and found the process manageable and useful  CRGs faced several issues when making judgements, and we adjusted guidelines and guidance  CRGs varied in the amount of time they took to classify Cochrane Reviews Next steps  Consider whether framework is suitable for non-intervention reviews  User test framework on readers of CDSR, with mock-ups of potential views  Implementation

23.  How should Cochrane Reviews be classified if there is more than one comparison, and the comparisons have different categories?  How should a Cochrane Review be classified if trials are ongoing?  Could there be more subdivisions in the categories, for example, to highlight whether an author is or isn’t available?  When should a Cochrane Review be classified as up to date? If there are no new studies, can the Cochrane Review be labelled ‘Current, Up to date’? And what should happen if a few small studies are identified that are unlikely to change the conclusions?  Is the term ‘Current, Up to date’ misleading when a full systematic update search as not been done?  How frequently should these statuses be applied?  For the classification ‘Current, No update intended’, how should CRGs make this judgement?  Should there be a final search before a Cochrane Review is classified as ‘Historical, No update intended’.  Should authors be consulted before making a judgement about the classifications?

26. Recommendations “Prioritise Cochrane Reviews for updating at least every two years, using methods such as the Updating prioritisation tool or the Updating decision tool...” “Replace the current guidance to update all Cochrane Reviews every two years, in favour of prioritising updates (see above)...” “Ensure decisions around prioritisation are transparent to readers.” “Classify Cochrane Reviews of interventions using the classification framework, at least every two years...”

27. Working groups Group 1: “Prioritise Cochrane Reviews for updating at least every two years, using methods such as the Updating prioritisation tool or the Updating decision tool...” AND “Ensure decisions around prioritisation are transparent to readers.” Group 2:“Replace the current guidance to update all Cochrane Reviews every two years, in favour of prioritising updates (see above)...” Group 3: “Classify Cochrane Reviews of interventions using the classification framework, at least every two years...”

28. To consider in working groups Agree/disagree? Consensus? If agreement, high or low priority (compared with other strategic session paper recommendations)? How urgent? If disagreement, tweak, rework, or discount? Implementation? – Changes to tools or framework? – Affect on CRGs’ monitoring? – Changes to Review Manager and Archie? – Changes to The Cochrane Library website? – Changes to CRG entity websites? – Changes to resources (e.g. Cochrane Handbook)? Barriers? – Methods? – External perceptions?