1. Meta-Analysis for Clinical Researchers An Introduction to Systematic Reviews & Meta-analysis

2. Topics Methods for pooling evidence across independent studies Pooling binary and continuous outcomes Differences between fixed and random effects models Guidelines for appraising published systematic reviews/meta- analyses

3. Why Meta-analysis/Systematic Reviews? “... the mass of new information makes it difficult for practicing physicians to follow the literature in all areas that might be relevant to their practices. New methods to synthesize and present information from widely dispersed publications are needed....” Jerome Kassirer. Clinical trials and meta- analysis: what do they do for us? N Engl J Med 1992; 327:273-4.

4. Why Need Meta-analysis? Information Explosion 10-fold Increase in Number of Professional Journals Psychology Journals: 91 (1951) --> 1,175 (1992) Math Science Journals: 91 (1953) --> 920 (1992) Biomedical Journals: 2,300 (1940)--> 23,000 (1993)

5. Problem – Conflicting Information Not only is there more information, but... Not all information is of equal quality Information does not necessarily = evidence There is often conflicting information & reports Traditional narrative reviews can be very “impressionistic”

6. Problems With Traditional Literature Reviews Addressed in Meta-analysis Selective inclusion of studies, often based on the reviewer's own impressionistic view of the quality of the study Differential subjective weighting of studies in the interpretation of a set of findings Misleading interpretations of study findings Failure to examine characteristics of the studies as potential explanations for disparate or inconsistent results across studies Failure to examine moderating variables in the relationship under examination

7. Brief History of Meta-analysis (very abbreviated) 1900’s: Karl Pearson pooled correlation coefficients - enteric fever & inoculation rates British Army 1930’s: Ronald Fisher pooled p-values 1972: Richard Peto: log rank test for combining (binary) data from different trials; Mantel, Thomas Chalmers 1976: Gene Glass “meta-analysis” effects of psychotherapy 1989 Murray Enkin, Marc Keirse, Iain Chalmers Effective Care in Pregnancy & Childbirth 1992/1993 UK Cochrane Centre/Cochrane Collaboration created

8. Primary, Secondary and Meta- analysis of Research “Primary analysis is the original analysis of data in a research study.... Secondary analysis is the re-analysis of data for the purpose of answering the original research question with better statistical techniques, or answering new questions with old data....” GV Glass 1976, p. 3

9. Primary, Secondary and Meta- analysis of Research “Meta-analysis refers to the analysis of analyses.... the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies which typify our attempts to make sense of the rapidly expanding research literature.” GV Glass 1976, p. 3

10. Rationale for Systematic Reviews “provide summaries of what we know, and do not know, that are as free from bias as possible.” (Chalmers et al 1999) “research that uses explicit & transparent methods to synthesise relevant studies, allowing others to comment on, criticise or attempt to replicate the conclusions reached. Systematic reviews follow same set of procedures as any individual study, & are often reported in the same way....” (Petrsino et al 1999)

11. 4 Basic Questions That a SR/MA Tries to Answer Are the results of the different studies similar? To the extent that they are similar, what is the best overall estimate of effect? How precise and robust is this estimate? Can dissimilarities be explained? Lau J, Ioannidis JPA, Schmid CH. Quantitative Synthesis in Systematic Reviews. Annals of Internal Medicine 1997; 127:820-826.

12. What is a Systematic Review? State objectives of the review, & outline eligibility criteria Search for studies that seem to meet eligibility criteria Tabulate characteristics of each study identified & assess its methodological quality Apply eligibility criteria & justify any exclusions

13. What is a Systematic Review? Assemble the most complete dataset feasible, with involvement of investigators Analyse results of eligible studies. Use statistical synthesis of data (meta-analysis) if appropriate & possible Perform sensitivity analyses, if appropriate & possible (including subgroup analyses) Prepare a structured report of the review, stating aims, describing materials & methods, & reporting results

14. Cochrane Collaboration Comparison with human genome project in potential impact for clinical medicine Naylor CD. Grey zones of clinical practice: some limits to evidence-based medicine. Lancet 1995; 345:840-2. What is the Cochrane Collaboration & why is it important?

15. Cochrane Library CD (& WWW) Cochrane Database of Systematic Reviews (CDSR) Database of Abstracts of Reviews of Effectiveness (DARE) Cochrane Central Register of Controlled Trials (CENTRAL) Cochrane Review Methodology Database Health Technology Assessment DB (HTA) NHS Economic/Evaluation Database (NHS EED) Cochrane Library CD (& WWW)

16. Sources of Evidence in “Real-time”: Cochrane Library (Issue 1, 2003) CDSR (Cochrane Database of Systematic Reviews) –>50 CRGs, >1500 complete reviews, >1200 protocols, >200 new reviews/year DARE (DB of Abstracts of Reviews of Effectiveness) >3800 abstracts Cochrane Central Register of Controlled Trials (CENTRAL) >353,000 RCTs/CCTs HTA (Health Technology Assessment Database) >2800 NHS Economic /Evaluation DB >10,000

17. Retrieval Problems –50% of RCTs not found in MEDLINE –44% of all RCTs (17% - 76%) 72% of RCTs in MEDLINE (32% - 91%) –Publication Type (PT) indexing –RCT (1991) –CCT (1995) –Meta-analysis (1993)

18. Hand Searching Medical Journals >630 journals 19,266 RCTs tagged (1991-93) 14,964 CCTs tagged (1985-90) MEDLINE updated annually to include or re-tag these studies as RCTs or CCTs

19. Distinguishing Feature of Cochrane Reviews “The fundamental distinction between Cochrane Reviews and other reports of systematic reviews is that their authors are expected to update and amend them in the light of relevant additional data, and criticisms or other comments.” Sir Iain Chalmers 1999

20. Example of a Cochrane Systematic Review/Meta-analysis Crowley P. Prophylactic corticosteroids for preterm birth (Cochrane Review). In: The Cochrane Library, Issue 2, 2003.

21. Effects of Asthma Self Management Educational Interventions in Children and Adolescents: A Systematic Review and Meta-analysis James P Guevara, MD,MPHa, Fredric M Wolf, PhDb, Cyril M Grum, MDc, & Noreen M Clark, PhDc aUniversity of Pennsylvania, Philadelphia, USA bUniversity of Washington, Seattle, USA cUniversity of Michigan, Ann Arbor, USA Cochrane Library 2003 (1); BMJ 2003; in press.

22. Objectives/Purpose To determine the effectiveness of asthma self- management education programs on health outcomes in children Outcomes: Lung Function Asthma Morbidity Heath Care Utilization: Self-reported Perceptions of Self-care Abilities

23. Objectives/Purpose –To conduct subgroup analyses to examine the impact of –type of educational intervention (individual vs. group) –intensity of educational intervention (no. of sessions) – self-management strategy (symptom vs. peak flow) –degree of asthma severity –length of follow-up –study quality (adequacy of allocation concealment, withdrawal rates, and type of trial, i.e., RCT vs. CCT)

24. Inclusion Criteria for Studies Randomized controlled trials (RCTs) or controlled clinical trials (CCT) Children & adolescents ages 2 to 18 years old Educational intervention designed to teach one or more self-management strategies related to prevention, attack management, or social skills Included outcomes on pulmonary function tests, morbidity, or health care utilization

25. Search Strategy – References & Databases Studies were identified from –Cochrane Airways Group's Special Register of Controlled Trials comprised of references from –MEDLINE (1966-2000) –EMBASE (1980-2000) –CINAHL (1982-2000) hand searched airways-related journals PsychINFO Reference lists from relevant review articles that were identified (ancestry approach)

26. Search Strategy - Terms asthma OR wheez* AND education* OR self management OR self- management AND placebo* OR trial* OR random* OR double-blind OR double blind OR single- blind OR single blind OR controlled study OR comparative study.

27. Identification of Trials Potentially relevant studies from literature search and hand searches (n=318) Excluded on basis of abstract, e.g., not randomised or controlled clinical trials (n=273) Articles selected for full text review (n=45) Excluded after full text review (n=13) Eligible trials (n=32)

28. Main Outcome Measures –Lung Function (pulmonary function tests): –FEV1 PEFR (Peak Flow ) –Asthma Morbidity: –asthma exacerbations –days of school absence –days of restricted activity –nights disturbed by asthma –asthma severity

29. Main Outcome Measures –Health Care Utilization: –physician visits –emergency department (ED) visits hospitalizations –Self-reported Perceptions of Self-care Abilities: –self-efficacy

30. Subgroup Analyses Time of enrollment in the intervention (1-6 mo, 7-12 mo, or >/=12 months) Self- management strategy (peak flow-based vs. symptom-based) Intervention type (individual vs. group) Intensity of intervention (single vs. multiple sessions) Study quality (RCT vs. CCT; random allocation procedures)

31. Sources of Bias RCTs (primary studies) –Selection bias –Performance bias –Exclusion bias –Detection bias Meta-analyses –Publication bias –Language bias –Coder bias “Apples & oranges” vs “fruit” (heterogeneity) –Quality bias (small studies) –Multiple publication bias

32. 32 An Inverted Funnel Plot to Detect Publication Bias

33. 33 An Inverted Funnel Plot to Detect Publication Bias

34. Pooling Binary & Continuous Outcomes –General principals –Effect size –Confidence Intervals –Types of data –Sources of potential bias Interpreting results Examples

35. Differences Between Fixed & Random Effects Models So what’s in a model? Why does it matter? How to deal with heterogeneity?

36. Heterogeneity Common, to be expected, not the exception Should do test for homogeneity, but... interpret heterogeneity cautiously in spirit of exploratory data analysis –Exploring sources of heterogeneity can lead to insights about modification of apparent associations by various aspects of –Study design –Exposure measurements –Study populations

37. Heterogeneity Relations discovered in process of exploring heterogeneity may be useful in planning & carrying out new studies Excluding outliers solely on basis of disagreement with other studies can lead to seriously biased summary estimates (avoid) Easier to interpret sources of heterogeneity when identified in advance of data analysis (not when suggested only by data)

38. Fixed & Random Effects Fixed effects models assume that an intervention has a single true effect Random effects models assume that an effect may vary across studies

39. Random Effects Assumes sample of studies randomly drawn from population of studies This is NOT typically true because: –All trials are included –Trials are systematically (e.g., conveniently) sampled and not randomly sampled

40. Random Effects Primary value of M-A is in search for predictors of between-study heterogeneity Random-effects summary is last resort only when predictors or causes of between-study heterogeneity cannot be identified Random-effects can conceal fact that summary estimate or fitted model is poor summary of the data Sander Greenland. Am J Epidemiol 1994;140;290-6.

41. Random Effects Sometimes needed, but more sensitive to publication bias than fixed-effects Random effects weights vary less across studies than fixed-effects weights W = 1/v versus w = 1/(v + t2) Leads to reduced variation in weights Thus smaller studies given larger relative weights when random effects models used Thus influenced more strongly by any tendency NOT to publish small statistically insignificant studies  biased estimate, spuriously strong associations

42. Random Effects Fixed effects weights vs. random effects weights W = 1/v versus w = 1/(v + t2) Identical when there is little or no between study variation When differ, confidence intervals are larger for random-effects than fixed effects Smaller studies given larger relative weights in random effects models & > influence Conversely, influence of larger studies is less May result in type II (beta error), e.g., Finding no significant difference when one truly exists

43. 43 Nights Disturbed by Asthma 3 trials, n = 202 (SMD –0.34, 95% CI –0.62 to –0.05) fixed effects (SMD –0.39, 95% CI –1.07 to +0.28) random effects

44. 44 Methodologic Choices & Their Implications in Dealing With Heterogeneous Data in a Meta-analysis Lau J, Ioannidis JPA, Schmid CH. Quantitative Synthesis in Systematic Reviews. Annals of Internal Medicine 1997; 127:820-826.