How Empty Are Empty Reviews? The first report on the Empty Reviews Project sponsored by the Cochrane Opportunities Fund and an invitation to participate in the development of reporting guidelines Joanne Yaffe, PhD, University of Utah Paul Montgomery, PhD, University of Oxford Sally Hopewell, PhD, UK Cochrane Centre Lindsay Shepard, MSc, University of Utah The Empty Reviews Project is funded by the Cochrane Opportunities Fund

Summary of Presentation The Problem of Empty Reviews Current State of Cochrane Empty Reviews and Apparent Direction of Travel How this Project Aims to Proceed Cochrane Contributor Inputs The Cochrane Empty Reviews Project

Introduction: Why study empty reviews? First, what do we mean by Empty Review? An empty review is a systematic review that reports no studies eligible for inclusion. The Cochrane Empty Reviews Project

Lack of high quality studies to address question posed. – Very narrow population – Very focused intervention – Outcomes that are not measured. Many plan empty reviews to justify new trials in response to funding policies. The Cochrane Empty Reviews Project How do systematic reviews come to be empty?

Policy-makers and practitioners report frustration in locating a review only to find that it has no included studies and is of limited utility for clinical decisions. Should observations be made based on excluded studies to provide decision-makers with some guidance? (following Lang, Edwards & Flieszer 2007) The Cochrane Empty Reviews Project

The Cochrane Handbook (2009) does not address the reporting of empty reviews specifically, although it does address excluded studies in general. The Cochrane Empty Reviews Project

“A Cochrane review includes a list of excluded studies, detailing any studies that a reader might plausibly expect to see among the included studies. This covers all studies that may on the surface appear to meet the eligibility criteria but on further inspection do not, and also those that do not meet all of the criteria but are well known and likely to be thought relevant by some readers. By listing such studies as excluded and giving the primary reason for exclusion, the review authors can show that consideration has been given to these studies. The list of excluded studies should be as brief as possible…” The Cochrane Empty Reviews Project Excluded Study Guidance in Cochrane Handbook (p. 154)

Concern: Empty reviews may leave decision-makers dependent on weaker research which may be presented without sufficient discussion of limitations. Concern: Empty reviews may fail to provide any implications either for practice or research. The Cochrane Empty Reviews Project

Empty Reviews Project Aim: To develop guidelines, in consultation with stakeholders and the editors of the Cochrane Handbook, for authors of reviews who do not find studies eligible for inclusion. The Cochrane Empty Reviews Project

Prevalence of Empty Reviews in The CDSR (as of 15 August 2010) Total reviews in The CDSR N= 4320 Total empty reviews in The CDSR N= 375 (8.7%) The Cochrane Empty Reviews Project

Number of Empty Reviews across CRGs The Cochrane Empty Reviews Project

Proportion of Empty Reviews across CRGs The Cochrane Empty Reviews Project

Empty Reviews by Year of First Publication The Cochrane Empty Reviews Project

Age of Empty Reviews in The CDSR Half of all empty reviews are 3 years or older, with some dating to Mean Age = 3.3 years (SD 3.4) The Cochrane Empty Reviews Project

Time since last update of Empty Revi ews 48.3% of all empty reviews have been updated within the last 2 years, but 5.9% (21) have not received an update in 7 years or longer. Mean Time since Update = 2.8 years (SD 2.2) The Cochrane Empty Reviews Project

How many excluded studies are listed in these empty reviews? 95 (25.3%) of empty reviews do not list excluded studies. The number of excluded studies ranges from 0 to 124. Mean number of excluded studies = 9.6 (SD 14.5) The Cochrane Empty Reviews Project

How many studies are listed as pending assessment? 360 (96.0%) of empty reviews do not list pending studies. The number of pending studies ranges from 0 to 9. Mean number of pending studies = 0.1 (SD 0.7) The Cochrane Empty Reviews Project

How many studies are listed as on-going? 338 (90.1%) of empty reviews do not list on-going studies. The number of on-going studies ranges from 0 to 4. Mean number of on-going studies = 0.1 (SD 0.5) The Cochrane Empty Reviews Project

How many reviews list no studies at all? 88 (23.5%) of empty reviews do not list any excluded, pending, or on-going studies. The Cochrane Empty Reviews Project

So what? So, what are the implications of all these empty reviews?  The lack of guidelines for reporting studies that have no included studies may result in inconsistently reported reviews.  Policy-makers and practitioners might not rely on Cochrane reviews for decision-making.  More importantly, however, the lack of guidelines for authors of empty reviews may result in less than useful conclusions and further, may misstate results. The Cochrane Empty Reviews Project

What do we mean by “less than useful conclusions”? The Cochrane Empty Reviews Project Less than useful conclusions.

What sorts of conclusions might be less than useful? Reference to single studies without appropriate caveats. Unsupported statements. No statements at all. The Cochrane Empty Reviews Project

Review A : Implications for practice Sufficient evidence based on randomised controlled trials is lacking and therefore we can draw no reliable conclusions about the comparative effectiveness of medical and surgical interventions of [deleted to de-identify]. However, the studies identified, but excluded from this review, suggest there may be a considerable response to placebo treatment in this condition. Review B: Implications for practice On the evidence currently available, [ ] cannot be recommended for the treatment of [ ]. Implications for research [ ] has shown promise in animal studies but there are no randomized double-blind placebo-controlled trials in people with [ ]. Such trials are needed. The Cochrane Empty Reviews Project

What might be better ways to handle conclusions? Specific recommendations for design of trials. Appeal to related reviews, drawing analogy to present question, with caveats. Balanced discussion of weaker studies. The Cochrane Empty Reviews Project

Review C: Implications for research 1.General As with all similar reviews, public registration of a study before anyone is randomised would ensure that participants could be confident that people would know that the study had at least taken place. Compliance with CONSORT (Moher 2001), both on the part of authors and editors, would help to clarify methodology and many outcomes. Failure to comply with CONSORT guidelines results in loss of data and confusion in results, neither of which helps clinicians, patients or managers Specific [ ] is a most distressing [ ] disorder that remains highly prevalent, both in the developed and developing world. This review highlights the need for well designed, conducted and reported clinical trials (Table 1) to address the claims of open studies as regards the effects of the [ ] group of drugs for [ ]. The Cochrane Empty Reviews Project

What do we hope to accomplish? Consistency in the way that excluded studies are addressed in the authors’ conclusions. Collaboration with other stakeholders towards development of guidelines for inclusion in The Cochrane Handbook. The Cochrane Empty Reviews Project

Our Plan 1.Complete our analysis of the 375 empty reviews in The CDSR as of 15 August, Solicit input from all interested parties via and (coming soon!). 3.Develop specific recommendations around the writing, presentation, and persistence in updating empty reviews. The Cochrane Empty Reviews Project

4.Present our findings and recommendations at a consensus meeting of key stakeholders in Spring, 2011, including Cochrane Handbook editors. 5.Use the results of the consensus meeting to refine our recommendations. 6.Present our findings to the Madrid Colloquium in October 2011, for further discussion. The Cochrane Empty Reviews Project

The Cochrane Empty Reviews Project Funded by The Cochrane Opportunities Fund Paul Montgomery, PhD, University of Oxford, PI Sally Hopewell, PhD, UK Cochrane Centre, Co-PI Joanne Yaffe, PhD, University of Utah, Co-PI Lindsay Shepard, MSc, University of Utah, Research Associate