1. gbiondizoccai@gmail.com www.metcardio.org Failures of meta-analyses: the evidence against the evidences Giuseppe Biondi-Zoccai, MD Division of Cardiology, Department of Internal Medicine, University of Turin, Turin, Italy Meta-analysis and Evidence-based medicine Training in Cardiology (METCARDIO), Turin, Italy

2. gbiondizoccai@gmail.com www.metcardio.org What brings you from this… Levi et al, Lancet 1999

3. gbiondizoccai@gmail.com www.metcardio.org …to this?

4. gbiondizoccai@gmail.com www.metcardio.org Index How to define meta-analyses? Key concepts What comes first? Scientific hierarchy and The Cochrane Collaboration Where’s the beef? Strenghts of meta-analyses Any toxic asset? When meta-analyses fail All that glitters is not gold. Appraising systematic reviews

5. gbiondizoccai@gmail.com www.metcardio.org Index How to define meta-analyses? Key concepts What comes first? Scientific hierarchy and The Cochrane Collaboration Where’s the beef? Strenghts of meta-analyses Any toxic asset? When meta-analyses fail All that glitters is not gold. Appraising systematic reviews

6. gbiondizoccai@gmail.com www.metcardio.org Why meta-analyses are important: exponential increase in pertinent PubMed citations on… PubMed search strategy: ("2001"[PDAT] : "2005"[PDAT]) AND (("systematic"[title/abstract] AND "review"[title/abstract]) OR ("systematic"[title/abstract] AND "overview"[title/abstract]) OR ("meta-analysis"[title/abstract] OR "meta-analyses"[title/abstract])) 14 4/2009 - PubMed query: (thrombosis OR hemostasis OR haemostasis OR coagulation OR clot* OR tranexamic OR aprotinin* OR rfviia OR desmopressin*) AND (meta-analy* OR metanalys* OR metaanalys* OR (systematic AND (review OR overview))) …thrombosis, hemostasis, coagulation, tranexamic acid, aprotinin, rFVIIa, or desmopressin

7. gbiondizoccai@gmail.com www.metcardio.org Why should you trust me? Meta-analyses or manuscript pertinent to meta- analyses that I have co-authored since graduation Total = 51

8. gbiondizoccai@gmail.com www.metcardio.org Famous quotes “If I have seen further it is by standing on the shoulders of giants” Isaac Newton “The great advances in science usually result from new tools rather than from new doctrines” Freeman Dyson

9. gbiondizoccai@gmail.com www.metcardio.org Famous quotes “I like to think of the meta- analytic process as similar to being in a helicopter. On the ground individual trees are visible with high resolution. This resolution diminishes as the helicopter rises, and in its place we begin to see patterns not visible from the ground” Ingram Olkin

10. gbiondizoccai@gmail.com www.metcardio.org Baby steps of meta-analysis 1904 - Karl Pearson (UK): correlation between inoculation of vaccine for typhoid fever and mortality across apparently conflicting studies 1931 – Leonard Tippet (UK): comparison of differences between and within farming techniques on agricultural yield adjusting for sample size across several studies 1937 – William Cochran (UK): combination of effect sizes across different studies of medical treatments 1970s – Robert Rosenthal and Gene Glass (USA), Archie Cochrane (UK): combination of effect sizes across different studies of, respectively, educational and psychological treatments 1980s – exponential development/use of meta-analytic methods

11. gbiondizoccai@gmail.com www.metcardio.org Minimal glossary Review: viewpoint on a subject quoting different primary authors Overview: as above Qualitative review: deliberately avoids a systematic approach Systematic review: deliberately uses a systematic approach to study search, selection, abstraction, appraisal and pooling Quantitative review: uses quantitative methods to appraise or synthesize data Meta-analysis: uses specific statistical methods for data pooling and/or exploratory analysis Individual patient data meta-analysis: uses specific stastistical methods for data pooling or subgroup exploration exploiting individual patient data →Our key goal: systematic review (± meta-analysis)

12. gbiondizoccai@gmail.com www.metcardio.org Qualitative review Elwood et al, Lancet 2009

13. gbiondizoccai@gmail.com www.metcardio.org Systematic review and meta-analyses What is a systematic review? –A systematic appraisal of the methodological quality, clinical relevance and consistency of published evidence on a specific clinical topic in order to provide clear suggestions for a specific healthcare problem What is a meta-analysis? –A quantitative synthesis that, preserving the identity of individual studies, tries to provide an estimate of the overall effect of an intervention, exposure, or diagnostic strategy

14. gbiondizoccai@gmail.com www.metcardio.org Systematic review (w/o meta-analysis) Berntorp et al, Eur J Haematol Suppl 2008

15. gbiondizoccai@gmail.com www.metcardio.org Systematic review and meta-analysis Agostoni et al, J Am Coll Cardiol 2004

16. gbiondizoccai@gmail.com www.metcardio.org Index How to define meta-analyses? Key concepts What comes first? Scientific hierarchy and The Cochrane Collaboration Where’s the beef? Strenghts of meta-analyses Any toxic asset? When meta-analyses fail All that glitters is not gold. Appraising systematic reviews

17. gbiondizoccai@gmail.com www.metcardio.org EBM hierarchy of evidence 1.N of 1 randomized controlled trial 2.Systematic reviews of homogeneous randomized trials 3.Single (large) randomized trial 4.Systematic review of homogeneous observational studies addressing patient-important outcomes 5.Single observational study addressing patient-important outcomes 6.Physiologic studies (eg blood pressure, cardiac output, exercise capacity, bone density, and so forth) 7.Unsystematic clinical observations Guyatt and Rennie, Users’ guide to the medical literature, 2002

18. gbiondizoccai@gmail.com www.metcardio.org Parallel hierarchy of scientific studies in cardiovascular medicine Biondi-Zoccai, Ital Heart J 2003 Qualitative reviews Systematic reviews Meta-analyses from individual non- randomized studies Meta-analyses from individual patient data from randomized studies Case reports and series Observational studies Observational controlled studies Randomized controlled trials Multicenter randomized controlled trials Meta-analyses from individual randomized studies

19. gbiondizoccai@gmail.com www.metcardio.org

20. gbiondizoccai@gmail.com www.metcardio.org The Cochrane Collaboration Mission Statement: The Cochrane Collaboration is an world-wide organization that aims to help people make well­informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions

21. gbiondizoccai@gmail.com www.metcardio.org The Cochrane Collaboration Over 6000 contributors & over 5000 reviews 50 Collaborative Review Groups (CRGs) 12 centers throughout the world 9 fields 11 Methods Groups 1 Consumer Network The Campbell Collaboration (focusing on education/social sciences)

22. gbiondizoccai@gmail.com www.metcardio.org The Cochrane Collaboration Over 6000 contributors & over 5000 reviews 50 Collaborative Review Groups (CRGs) 12 centers throughout the world 9 fields 11 Methods Groups 1 Consumer Network The Campbell Collaboration (focusing on education/social sciences) Thus currently the best reviews available (but still far from perfection)!

23. gbiondizoccai@gmail.com www.metcardio.org Index How to define meta-analyses? Key concepts What comes first? Scientific hierarchy and The Cochrane Collaboration Where’s the beef? Strenghts of meta-analyses Any toxic asset? When meta-analyses fail All that glitters is not gold Appraising systematic reviews

24. gbiondizoccai@gmail.com www.metcardio.org Pros Application to any clinical research question Systematic searches for clinical evidence Explicit and standardized methods for search and selection of evidence sources Thorough appraisal of the internal validity of primary studies Quantitative synthesis with increased statistical power Increased external validity by appraising the effect of an intervention (exposure) across different settings Test subgroup hypotheses (eg with patient-level reviews) Explore clinical and statistical heterogeneity Lau et al, Lancet 1998

25. gbiondizoccai@gmail.com www.metcardio.org Any application feasible: meta-analysis of intervention studies Landoni et al, Am J Kidney Dis 2006

26. gbiondizoccai@gmail.com www.metcardio.org Any application feasible: meta-analysis of diagnostic studies Hamon et al, JACC 2006

27. gbiondizoccai@gmail.com www.metcardio.org Any application feasible: meta-analysis of prognostic studies Biondi-Zoccai et al, Eur Heart J 2006

28. gbiondizoccai@gmail.com www.metcardio.org Thorough appraisal of internal validity and quality of selected studies Landoni et al, J Cardiothorac Vasc Anesth 2007

29. gbiondizoccai@gmail.com www.metcardio.org Increasing statistical power and external validity Agostoni et al, J Am Coll Cardiol 2003

30. gbiondizoccai@gmail.com www.metcardio.org Explore small study effects Abbate et al, J Am Coll Cardiol 2008

31. gbiondizoccai@gmail.com www.metcardio.org Index How to define meta-analyses? Key concepts What comes first? Scientific hierarchy and The Cochrane Collaboration Where’s the beef? Strenghts of meta-analyses Any toxic asset? When meta-analyses fail All that glitters is not gold. Appraising systematic reviews

32. gbiondizoccai@gmail.com www.metcardio.org Reasons for meta-analysis failure Duplicate efforts may lead to discordant results Funding or conflicts of interest may bias Studies/events might not be found Studies may be of low quality/internal validity Studies may be heterogeneous/inconsistent, ie “mixing apples with oranges” provides unreal fruits Studies may not be relevant to current individual practice Selection based on publication may bias Analysis with highly sensitive but unrobust tests may bias LeLorier et al, New Engl J Med 1997; Lau et al, Lancet 1998; Rosen, BMC BMC Health Services Research 2009

33. gbiondizoccai@gmail.com www.metcardio.org What if meta-analyses disagree? Biondi-Zoccai et al, BMJ 2006

34. gbiondizoccai@gmail.com www.metcardio.org What if there is financial bias? Barnes et al, JAMA 1998

35. gbiondizoccai@gmail.com www.metcardio.org What if there low quality studies are included? Henry et al, CDSR 2009; Rosen, BMC BMC Health Services Research 2009

36. gbiondizoccai@gmail.com www.metcardio.org What if there low quality studies are included? Henry et al, CDSR 2009

37. gbiondizoccai@gmail.com www.metcardio.org What if I mix apples and oranges… Hsia et al, Ann Surg 2008

38. gbiondizoccai@gmail.com www.metcardio.org What if I mix apples and oranges… Hsia et al, Ann Surg 2008 P for effect Incosistency P for heterogeneity

39. gbiondizoccai@gmail.com www.metcardio.org What if small positive studies are selectively published? Precision (standard error of log relative risk) Effect (relative risk) P<0.001 at Egger test P<0.001 at Peters test 0.01 0.1 1 10 100 0.0 0.4 0.8 1.2 1.6 Favours cilostazol Favours control Biondi-Zoccai et al, Am Heart J 2008

40. gbiondizoccai@gmail.com www.metcardio.org Index How to define meta-analyses? Key concepts What comes first? Scientific hierarchy and The Cochrane Collaboration Where’s the beef? Strenghts of meta-analyses Any toxic asset? When meta-analyses fail All that glitters is not gold. Appraising systematic reviews

41. gbiondizoccai@gmail.com www.metcardio.org Internal validity of primary studies Many scales for the quality of included studies have been reported, but none is reliable or robust The recommended approach is to individually appraise the potential risk of the 4 biases (eg A-low, B-moderate, C-high, D-unclear from reported data): –Selection bias (one group is different than the other) –Performance bias (treatment is systematically different) –Adjudication bias (outcome adjudication is selectively different) –Attrition bias (follow-up duration or completeness is different)

42. gbiondizoccai@gmail.com www.metcardio.org Another common classification scheme for bias

43. gbiondizoccai@gmail.com www.metcardio.org Appraisal tools: QUOROM Moher et al, Lancet 1999

44. gbiondizoccai@gmail.com www.metcardio.org Appraisal tools: QUOROM Moher et al, Lancet 1999

45. gbiondizoccai@gmail.com www.metcardio.org Appraisal tools: QUOROM Moher et al, Lancet 1999

46. gbiondizoccai@gmail.com www.metcardio.org Appraisal tools: Oxman and Guyatt’s Evaluates the internal validity of a review on 9 separate questions for which 3 distinct answers are eligible (“yes”, “partially/can’t tell”, “no”): Oxman et al, J Clin Epidemiol 1991 1. Where the search methods used to find evidence stated? 2. Was the search for evidence reasonably comprehensive? 3. Were the criteria for deciding which studies to include in the overview reported? 4. Was bias in the selection of studies avoided? 5. Were the criteria used for assessing the validity of the included studies reported? 6. Was the validity of all studies referred to in the text assessed using appropriate criteria?

47. gbiondizoccai@gmail.com www.metcardio.org Appraisal tools: Oxman and Guyatt’s 7. Were the methods used to combine the findings of the relevant studies reported? 8. Were the findings of the relevant studies combined appropriately relative to the primary question the overview addresses? 9. Were the conclusions made by the author(s) supported by the data and/or analysis reported in the overview? 10. This summarizes the previous ones and, specifically, asks to rate the scientific quality of the review from 1 (being extensively flawed) to 3 (carrying major flaws) to 5 (carrying minor flaws) to 7 (minimally flawed). If the “partially/can’t tell” answer is used one or more times in questions 2, 4, 6, or 8, a review is likely to have minor flaws at best and is difficult to rule out major flaws (ie a score≤4). If the “no” option is used on question 2, 4, 6 or 8, the review is likely to have major flaws (ie a score≤3). Oxman et al, J Clin Epidemiol 1991

48. gbiondizoccai@gmail.com www.metcardio.org A few references Biondi-Zoccai GGL et al. Parallel hierarchy of scientific studies in cardiovascular medicine. Ital Heart J 2003; 4: 819-20 Biondi-Zoccai GGL et al. Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study. BMJ 2006;332:202-209 Biondi-Zoccai GGL et al. A practical algorithm for systematic reviews in cardiovascular medicine. Ital Heart J 2004;5:486 -7 Bucher HC et al. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol 1997;50:683– 9 Cappelleri JC et al. Large trials vs meta-analysis of smaller trials: how do their results compare? JAMA 1996; 276: 1332-8 Clarke M et al, eds. Cochrane reviewers’ handbook 4.2.0. (www.cochrane.org/resources/handbook/handbook.pdf) Cooper H et al, eds. The handbook of research synthesis. New York, NY: Russell Sage Foundation, 1994 Cucherat M et al. EasyMA: a program for the meta-analysis of clinical trials. Comput Methods Programs Biomed 1997;53:187- 90 Egger M et al, eds. Systematic reviews in health care: meta-analysis in context. 2nd ed. London: BMJ Publishing Group, 2001 Glass G. Primary, secondary and meta-analysis of research. Educ Res 1976;5:3-8 Glasziou P et al. Systematic reviews in health care. A practical guide. Cambridge: Cambridge University Press, 2001 Guyatt G et al, eds. Users’ guides to the medical literature. A manual for evidence-based clinical practice. Chicago, IL: AMA Press, 2002 Higgins JPT et al. Measuring inconsistency in meta-analyses. BMJ 2003;327:557 – 60 Lau J et al. Summing up evidence: one answer is not always enough. Lancet 1998;351:123 -7 Moher D et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUORUM statement. Lancet 1999; 354: 1896-900 Petitti DB. Meta-analysis, decision analysis, and cost-effectiveness analysis: methods for quantitative synthesis in medicine. New York, NY: Oxford University Press, 2000 Song F et al. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analysis. BMJ 2003;326:472 Thompson SG et al. How should meta-regression analyses undertaken and interpreted? Stat Med 2002;21:1559-73

49. gbiondizoccai@gmail.com www.metcardio.org Take home messages

50. gbiondizoccai@gmail.com www.metcardio.org Take home messages The validity of a meta-analysis refers to the soundness of the original studies and the procedures used to combine them Dozens of potential validity threats have been identified, and should always be borne in mind Given its current pivotal role in the hierarchy of clinical evidence, all clinical decision-makers should have a working knowledge of how to appraise them

51. gbiondizoccai@gmail.com www.metcardio.org Take home messages The three rules of thumb to decide whether a meta-analysis can be trusted are: Were the included studies all based on proper randomization? Were the included studies clinically and statistically homogeneous? Are there at least 100 event in any of the two treatment groups for the end-point of interest?

52. gbiondizoccai@gmail.com www.metcardio.org Thank you for your attention! For any correspondence: gbiondizoccai@gmail.com For further slides on these topics feel free to visit the metcardio.org website: http://www.metcardio.org/slides.html gbiondizoccai@gmail.com http://www.metcardio.org/slides.html