New methodological standards for Cochrane reviews first output from the MECIR project edit Rachel Churchill Co-ordinating Editor representative on Steering Group School of Social and Community Medicine, University of Bristol Julian Higgins Methods Groups representative on Steering Group MRC Biostatistics Unit, Cambridge, UK & Centre for Reviews and Dissemination, University of York

Outline Background and purpose of MECIR Development of methodological standards for conduct of intervention reviews The first set of standards Some examples Next steps and discussions around implementation

Background Coordinated by MARS (Methods Application and Review Standards) – Co-convened by Julian Higgins and Rachel Churchill to facilitate interaction between Methods Groups and CRGs; supports Editor in Chief and CEU Coordinating Group from MARS and the CEU (Rachel Churchill, Julian Higgins, Jackie Chandler, David Tovey and Toby Lasserson)

MECIR Stands for Methodological Expectations of Cochrane Intervention Reviews Purpose is –to specify methodological expectations for Cochrane protocols, reviews, and updates of reviews on the effects of interventions –to ensure that these methodological expectations are supported and implemented across the Collaboration

Development process MECIR Working Groups 1.Developing a question and deciding the scope of the review 2.Searching for studies 3.Selecting studies and collecting data 4.Assessing risk of bias in studies 5.Analysing data and undertaking meta-analyses 6.Interpretation and presenting results Coordinating Group 1. WGs submitted long list of items 2. WGs asked to consider Mandatory items vs Highly Desirable vs Good practice and Rationale for each Collaboration-wide consultation All registered entities (16/6/11 – 8/8/11) 51% responded - CRGs (35/53) - MGs (8/15) - Centres (5/14) - Centre branches (2/15) - Fields (5/11) - CEU, TWG, Author rep

Consultation process Further work by Coordinating Group, before extensive consultation and feedback from all Cochrane entities All registered entities (16 June ‘11 – 8 August ‘11) 51% entities responded –CRGs (35/53) –MGs (8/15) –Centres (5/14) –Centre branches (2/15) –Fields (5/11) –CEU, TWG, Author rep

Consultation feedback (Aug – Oct 2011) High levels of support for most items Decision around status item description, expectation and rationale (merging or omitting as appropriate) and provided associated Handbook reference for each standard 100 standards, reduced to 80 Provided responses to comments and final decision regarding status, description, and rationale for each item (see Annex 2)

Summary of MECIR standards 80 standards for the conduct of CIRs structured around: –Setting research question to inform the scope of the review –Setting the eligibility criteria for including studies in the review –Selecting outcomes to be addressed –Planning the review methods –Searching for studies –Selecting studies into the review –Collecting data from included studies –Assessing risk of bias in included studies –Synthesizing the results of included studies –Summarizing the findings –Reaching conclusions

Some examples

Example of one we got right

Pre-defining clear and unambiguous criteria for interventions and comparators (standard 7) (Mandatory) Agreed/Disagreed: CRGs: 30 / 0 Centres: 7 / 0 Fields: 4 / 0 Methods Groups: 6 / 0 Left unchanged Define in advance the eligible interventions and the interventions against which these can be compared in the included studies

Example of one we had to work on

Excluding studies without useable data (standard 40) [Draft] (Mandatory) Agreed/Disagreed: CRGs: 23 / 5 Centres: 6 / 1 Fields: 4 / 0 Methods Groups: 6 / 0 Three suggestions to change to Highly Desirable Do not exclude studies on the basis of there being no usable data (either because the data are inconvenient format or because they are not reported at all)

Excluding studies without useable data (standard 40) Hugely important Clarify that this relates to the formal “exclusion” of studies from the review We will not implement this in our group RevMan will protest if a study is ‘included’ with no data at all agree if the outcomes of interest were measured as an outcome somewhat confusing should be rephrased

Excluding studies without useable data (standard 40) (Mandatory) Include studies in the review irrespective of whether measured outcome data are reported in a ‘usable’ way.

Some key ones

Some key conduct standards Consider important potential adverse effects: Mandatory Clarify role of outcomes: Mandatory Search trials registers: Mandatory Double (independent) data extraction: Mandatory Risk of bias tables: Mandatory Summary of findings tables: Highly desirable The five GRADE considerations (study limitations, consistency of effect, imprecision, directness and publication bias) to assess the quality of the body of evidence: Mandatory

The new stuff

Standards not currently in the Handbook Rerun or update searches for all relevant databases within six months before publication of the review or review update, and screen the results for potentially eligible studies: Mandatory, BUT Incorporate fully any studies identified in the rerun or update of the search within six months before publication of the review or review update: Highly Desirable Compare magnitude and direction of effects reported by studies with how they are presented in the review, taking account of legitimate differences: Mandatory Consider in advance whether issues of equity and relevance of evidence to specific populations are important...: Highly desirable

Other MECIR outputs Guidance on implementation of standards for updates to be produced –to be produced Reporting standards (for protocols, reviews, updates) –in draft –Working Groups to be invited to input –consultation: Collaboration and external (users) Good practices and Common errors –in draft –will continually evolve

Facilitating implementation A mixture of Distribution to entities and individuals Handbook amendments –prominent incorporation of standards throughout –extension/clarification of guidance as appropriate Training materials (Training Working Group) Check lists for authors, editors, peer reviewers –building on existing author checklist (Editorial Resources Committee) Modifications to RevMan Baseline audit

Concluding remarks MECIR conduct standards are (almost all) clarifications of existing Handbook guidance Adopting the standards should ensure that Cochrane intervention reviews are consistently of high quality We hope you will support and implement them –and contribute to development of further expectations, and future evaluations of them all The standards will be updated over time For more, see Finally, THANK YOU to all individuals and entities who have contributed so helpfully so far to the development of the expectations...

Particular thanks to Working Groups Doug Altman Mohammed Ansari (Methods lead, WG1) Sally Bell-Syer Patrick Bossuyt Deborah Caldwell Christopher Cates Rachel Churchill (Co-Eds lead, WG4) Mike Clarke (Co-Eds co-lead, WG2) Jan Clarkson (Co-Eds co-lead, WG6) Philippa Davies Marina Davoli (Co-Eds lead, WG1) Ruth Foxlee Chantelle Garritty Davina Ghersi (Co-Eds co-lead, WG2) Julie Glanville (Methods co-lead, WG2) Peter Herbison Julian Higgins Sophie Hill (Co-Eds lead, WG3) Toby Lasserson Edith Leclercq Carol Lefebvre (Methods co-lead, WG2) Jessie McGowan Rachel Marshall Ruth Mitchell Donal O’Mathuna Anna Noel-Storr Georgia Salanti (Methods lead, WG5) Doug Salzwedel Margaret Sampson Jelena Savovic Holger Schünemann (Methods lead, WG 6) Ian Shemilt Nandi Siegfried Jonathan Sterne (Methods lead, WG4) Britta Tendal (Methods lead, WG3) David Tovey Peter Tugwell Lucy Turner Claire Vale Julia Walters Helen Worthington (Co-Eds lead, WG 5 and Co-Eds co-lead, WG6) Janelle Yorke