1. Prospective Comparison between Rapid and Systematic Reviews on the Same Topics: A Feasibility Study
Junqiao Chen1, Jon Brassey2, Yanfang Ma3, Qi Wang4, Milesh Patel1, Samuel Boudreault5, Mark-Steven Howe6
1Evolent Health, Arlington, Virginia, USA, 2Trip Database, Newport, South Wales, UK, 3Chevidence Working Group, Lanzhou University, Lanzhou, Gansu, China, 4Evidence-Based Medicine Center and Chinese GRADE Center, Lanzhou University; Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou, Gansu, China, 5Family Medicine Department, Laval University, Québec City, Québec, Canada, 6Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, England, UK
June Oxford

3. Introduction Important to compare rapid review and systematic review
No widely accepted approach to do this in a large scale
“… research is extremely limited”.

4. Agenda A new study design piloted on two examples
Why it has good potential

5. Study design Systematic Review 1 Time Systematic Review 2
Rapid Review 1
Rapid Review 2
Rapid Review
Methodology
Result
Conclusion
Systematic Review
Methodology
Result
Conclusion
Compare
Apply
Apply
Real world use
Compare
Real world use
Four roles:
Selector
Main Reviewer
Second Reviewers
Assessors

6. Study design – random selection of systematic reviews
Selector
Main Reviewer
Second Reviewers
Assessors
Criteria:
Reviews on interventions
Narrow PICO
Not based on a previously published review …

7. Study design – Extraction of essential information
Selector
Main Reviewer
Second Reviewers
Assessors
Background sessions
Majority of the method sessions
Search strategy …
The goal: differences in results are due to “shortcuts” only

8. Study design – the conduct of rapid reviews (“ the rapidest”)
Selector
Main Reviewer
Second Reviewers
Assessors
PubMed only
English only
Shorter search look back (varied inter-reviews; fixed intra-reviews)
Single reviewer for screening, data extraction and quality appraisal

9. Study design – second reviewers
Selector
Main Reviewer
Second Reviewers
Assessors
*Document* errors
Do not change the rapid review manuscript.

10. Study design – external evaluation
Selector
Main Reviewer
Second Reviewers
Assessors
Read the rapid review first. Then answer:
How confident are you in using it in your clinical decision making? (0-10)
Any methodological limitations that are influencing your confidence?. (open question)
Read the systematic review. Then answer the same question.
Compare the two reviews:
Does the rapid review convey the essential information as the systematic one? (Y/N)
...

11. Result: the first review
P - Viral myocarditis I - Corticosteroids C - No corticosteroids O – Mortality (primary outcome only)

12. Result: the first review – search result
618 references
Change from sensitivity-maximizing version to
sensitivity- and precision-maximizing version of the RCT filter
Limit to 20-year look back
57 references*
4 trials included

13. Result: the first review – search result: why different?
8 Trials
1 trial has 3 references. Only 1 from PubMed*
4 Trials
3 trials from Chinese databases only
1 trial is a mistake
3 trials
1 trial might be a mistake**
* The one from PubMed doesn’t meet inclusion criteria: Not randomized
** Doesn’t meet inclusion criteria: Mortality was not pre-defined and reported.

14. Result: the first review – data extraction and quality assessment
No error found

15. Result: the first review – result
Rapid Review
“Quality of evidence is low”.
“Corticosteroids make no improvement to the long-term mortality”.
Systematic Review
“The trials were small and of poor quality”
“Mortality was not reduced with corticosteroids either for children or for adults”.

16. Result: the first review – external evaluation
How confident are you in using the review conclusion in your clinical decision making? (0-10)
Rapid Review
Systematic Review
GP and academics in UK 6 8
Pharmacist in USA
GP in Canada and academics in UK 10
Methodologist in USA

17. Result: the first review – external evaluation
Rapid Review
Systematic Review
GP and academics in UK1 6 8
Pharmacist in USA2
GP in Canada and academics in UK3 10
Methodologist in USA4
Expertise matters: “…I think the Cochrane review is more authoritative but I think this is probably because the authors has more clinical experience / wisdom in cardiology, probably not because the exhaustive search for RCTs was necessary”.
Length ≠ quality: “... there aren’t enough data to support a decision either way. For the systematic review, the authors went into much more detail but in the end my conclusion is the same. I hope quantity of words does not cover up the lack of quality represented in the studies...”
Unpublished data is needed for controversial treatment.
No review is perfect: “ …Both reviews have the same conclusions and methodological limitations. I would most likely never give a rating of 10 given the complex nature of the disease and comorbidity of these patients and possibility of clinical effect on some individuals... ”

18. Result: the second review
P - Children and adolescents I - Fluoride mouthrinses  C - Placebo or no treatment O - Preventing dental caries

19. Result: the second review - search
196 references
20-year look back
30-year look back
109 references*
161 references*
2 trials included
3 trials included
28 trials before 30-year look back

20. Result: the second review – search: why different?
37 Trials
3 trials in rapid review
28 trials published before 1986
2 trials excluded in rapid review by mistakes
2 trials in PubMed but can’t ben identified by search strategy
1 trial’s full text not available electronically
1 trial not in English

21. Result: the second review – result
Rapid Review
Quality is very low.
No effect
Systematic Review
Moderate quality.
Large reduction in caries increment

22. Result: the second review – external evaluation
British dentist
Confidence: 4 in rapid review versus 9 in systematic review
“The search strategy is too limited resulting in … studies from Nordic countries only, which makes no sense considering the number of International studies out there.”

23. Result: the second review – second reviewers
Found two minor errors in extracted data.
Found two potential errors in risk of bias assessments.

24. Discussion – this approach is feasible for wider use
Economical: main reviewer (beginner level), spare-time, 4 months
Able to detect when rapid review works and when doesn’t work
Able explain why there are differences
End-user oriented: not just comparing the “grand mean”, but “confidence” in real world use.

25. Discussion – future work
Validate the questions in external evaluation
Try different levels of acceleration

26. Conclusion
A prospective study design is proposed and piloted on two case studies.
It shows good potential.
Other researchers might adopt, test or modify this approach in their comparison.
Thank you!