1. Grant Stephen: Chair of the MBC Life Science Informatics Group & CEO, Tessella Inc: grant.stephen@tessella.com Creating Insight & Understanding from Scientific Data Life Science Informatics Group Introduction to Meta-Analysis Christopher H. Schmid, PhD Tufts-New England Medical Center 6 June 2008

2. The weight of medical knowledge Weight of the Index Medicus According to 10-Year Periods from 1879 to 1977

3.  Medicine requires hard evidence, often clinical trials  Evidence inconsistent across studies Different study populations Different treatments or protocols Quality of technical design or execution Random variation Introduction

4. Reasons for combining data Get an overall estimate of treatment effect Appreciate the degree of uncertainty Appreciate heterogeneity Forces you to think rigorously about the data

5. Meta-Analysis A scientific discipline which applies a protocol to critically evaluate and uses statistical methods to combine the results of (previous) research Provides a quantitative summary of the overall treatment effect (typically an overall estimate and confidence interval) Increasingly used to understand differences among studies to explain discrepancies of results and to generate hypotheses of interactions To guide future research

6. Some terms used for meta-analysis Systematic Review Overview Quantitative overview Research (evidence) synthesis Research Integration Pooling (implies lumping data altogether) Combining (implies performing procedures on data)

7. No one will criticize you for doing a systematic review. But as soon as you combine data, you will most likely have controversy

8. Why is that? Apples and oranges (heterogeneity) Garbage in, garbage out (quality) Selection of outcomes (soft or hard) Selection of studies Publication bias Assumptions used in quantifying results

9. Characteristics of Meta-Analyses  More heterogeneity than multicenter trial  Meta-analysis addresses random variation  Gives pooled estimate of treatment effect  Can be confirmatory or exploratory  Pooling may not be best solution  Need to explain variation

11. Systematic review protocol Well-focused study question Identification of studies (design, source, search strategy) Eligibility criteria (study, patient, and disease characteristics, treatments, outcomes) Data extraction (definition of outcomes, quality assessment) Data summary and analysis (outcomes, intention to treat)

12. Issues in formulating a question Narrow versus broad (for individuals/ subgroups or entire population) Scientifically meaningful and useful (based on sound biological and epidemiological principles) Very broadly defined questions may be criticized for mixing apples and oranges Narrowly focused questions have limited generalizability and may lead to biased conclusions

13. Randomized comparisons of antibiotics for acute sinusitis

14. Literature processing Questions Search strategy Screening of abstracts Retrieve potential articles Screen full articles Data extraction on qualifying articles

15. Issues in Finding and Retrieving Evidence Search strategies Sources Language selection Published vs. unpublished literature Use of abstracts File drawer problem (publication bias) Multiple publications on same subjects Disproportionate amount of data for topic

16. Types of Multiple Publications Overlapping data (preliminary and later reports) Same data but different authors Similar data (same authors) but different cohort –need to verify with authors

17. Most meta-analyses are retrospective exercises, suffering from all the problems of being an observational design. We cannot fix bad data.

18. Basic principles in combining data For each analysis, one study should contribute only one effect Effect may be single outcome or composite of several outcomes Effect being combined should be same or similar across studies

19. What kinds of control? No treatment control Placebo Active comparator

20. Types of data that could be combined dichotomous (events, e.g. deaths) measures (odds ratios, correlations) continuous data (mmHg, pain scores) survival curves diagnostic test (sensitivity, specificity) individual patient data “effect size”

21. Issues in choosing method to combine studies Metrics Fixed vs. random effects model Treatment effect heterogeneity Baseline rate heterogeneity Weight

22. Heterogeneity (diversity) Is it reasonable (are studies and effects sufficiently similar) to estimate an average effect? Types of heterogeneity –Conceptual (clinical) heterogeneity –Statistical heterogeneity

23. Conceptual (scientific) Heterogeneity Are the studies of similar treatments, populations, settings, design, etc., such that an average effect would be scientifically meaningful?

24. Statistical Heterogeneity Is the observed variability of effects greater than that expected by chance alone?

25. Meta-analytic approaches  Summary point estimation (random or fixed effect model)  Meta-regression - modeling aggregate data heterogeneity  Baseline risk meta-regression  Response surface - individual patients’ data analysis

26. Sensitivity analyses Exclude studies Analyze subgroups Change assumptions Use different metric Compare fixed versus random effects model Perform cumulative meta-analysis

28. Dichotomous outcomes Binary outcomes, event or no event, yes or no Most common type of outcome reported in clinical trials (about 70%) Examples: dead/alive, stroke/no stroke, cure/failure 2x2 tables commonly used to report their results Sometimes continuous variables are forced into dichotomous outcomes. –E.g., a threshold could be used for pain scores and reported as improved or not improved.

29. OR = (a*d) / (b*c)

30. Example of 2x2 table: ISIS-2 StreptokinasePlacebo Vascular deaths7911029 Survive78017566 TOTAL85928595

31. Available metrics for combining dichotomous outcome data Odds ratio (OR) Risk ratio (RR) Risk difference (RD) NNT (Number Needed to Treat) can be derived (inverse of the combined risk difference) = 1/RD

32. Calculating treatment effects in ISIS-2 StreptokinasePlacebo Vascular deaths7911029 Survive78017566 TOTAL85928595 RR = 0.0921 / 0.1197 = 0.77 OR = (791 x 7566) / (1029 x 7801) = 0.75 RD = 0.0921 – 0.1197 = -0.028 TR = 791/8592 = 0.0921CR = 1029 / 8595 = 0.1197

33. ISIS-2 vascular death estimate & 95% CI Estimate95% CI Risk ratio (RR)0.770.70 – 0.84 Odds ratio (OR)0.750.68 – 0.82 Risk difference (RD) -0.028-0.037 - -0.019 NNT (1/RD)3627 - 54 Streptokinase vs. Placebo

34. Same change in one scale may have different meaning in another scale TreatmentControl StudyEventsTotalRateEventsTotalRate Relative Risk Risk Difference A 100100010%200100020%0.510% B 110000.1%210000.2%0.50.1%

35. Effect of Small Changes (small numbers are fragile) Baseline case Effect of decrease of 1 event Effect of increase of 1 event Relative change of estimate 2/10 20% 1/10 10% 3/10 30% ± 50% 20/100 20% 19/100 19% 21/100 21% ± 5% 200/1000 20% 199/1000 19.9% 201/1000 20.1% ± 0.5%

36. Beta-Blockers after Myocardial Infarction - Secondary Prevention Experiment Control Odds 95% CI N Study Year Obs Tot Obs Tot Ratio Low High === ============ ==== ====== ====== ====== ====== ===== ===== ===== 1 Reynolds 1972 3 38 3 39 1.03 0.19 5.45 2 Wilhelmsson 1974 7 114 14 116 0.48 0.18 1.23 3 Ahlmark 1974 5 69 11 93 0.58 0.19 1.76 4 Multctr. Int 1977 102 1533 127 1520 0.78 0.60 1.03 5 Baber 1980 28 355 27 365 1.07 0.62 1.86 6 Rehnqvist 1980 4 59 6 52 0.56 0.15 2.10 7 Norweg.Multr 1981 98 945 152 939 0.60 0.46 0.79 8 Taylor 1982 60 632 48 471 0.92 0.62 1.38 9 BHAT 1982 138 1916 188 1921 0.72 0.57 0.90 10 Julian 1982 64 873 52 583 0.81 0.55 1.18 11 Hansteen 1982 25 278 37 282 0.65 0.38 1.12 12 Manger Cats 1983 9 291 16 293 0.55 0.24 1.27 13 Rehnqvist 1983 25 154 31 147 0.73 0.40 1.30 14 ASPS 1983 45 263 47 266 0.96 0.61 1.51 15 EIS 1984 57 858 45 883 1.33 0.89 1.98 16 LITRG 1987 86 1195 93 1200 0.92 0.68 1.25 17 Herlitz 1988 169 698 179 697 0.92 0.73 1.18

37. What is the average difference in DBP?

38. Simple Average (-6.2) + (- 7.7) + (-0.1) 3 = -4.7 mmHg -4.7 mmHg ____

39. Sample Size Weighted Average (554 x -6.2) + (304 x -7.7) + (39 x -0.1) 554 + 304 + 39 = -6.4 mmHg

40. General Weighted Average Effect Size where:d i = effect size of the i th study w i = weight of the i th study k = number of studies

41. Calculation of weights Generally the inverse of the variance of treatment effect Different formula for odds ratio, risk ratio, risk difference Readily available in books and software

42. Effect Size Dimensionless metric Combine standard deviations of diverse types of related effects Availability and selection of reported effects may be biased Variable importance of different effects Frequently used in education, social science literature Difficult to interpret results

43. Statistical Models of Pooling 2x2 Tables Fixed Effect Model Random Effect Model

44. David Bowers. Statistics from scratch. An introduction for Health Care Professionals. John Wiley & Sons, 1996.

45. Container with fixed number of white and black balls (fixed effects model)

46. Random sampling from container with fixed number of white and black balls (different sample size)

48. Principle - common truth Main Model - fixed effects weighted average Advantages - easy to interpret, applies to whole population Disadvantages - often simplistic; not applicable with heterogeneity Summary point estimation

49. Different containers with different proportions of white and black balls (Random effects model)

50. Random sampling from containers to get overall estimate of Ratio of white and black balls

53. Summary point estimation Principle - range of truth Main Model - random effects weighted average Advantages – realistic; allows between-study heterogeneity Disadvantages - less intuitive; no insight into heterogeneity

55. Chi-Square Homogeneity Test NOTE: d = ln(OR i d+ = ln(OR MH )w i = 1/variance (OR i ) Variance (OR i ) = 1/a i + 1/b i + 1/c i + 1/d i

56. Differences in fixed effects and random effects weights:Magnesium for AMI TreatmentControlOdds Ratio weight TrialNRateN Odds Ratio FEMREM ISIS-4290110.076290390.0721.0693.8%24.9% LIMIT-211590.07711570.1020.744.4%22.2% Rasmussen1350.0671350.170.350.5%12.4% Other 8 trials6870.0356640.080.411.3%40.5% Odds Ratio Q statistics for odds ratio = 31.2 (p=0.0006)1.05 P=0.52 0.59 P=0.009

58. What to do with meta-analyses when new RCTs appear?

59. Findings of Cumulative Meta-analysis  Clinical experts’ recommendations often are unreliably synchronized with developing RCT evidence.  Large clinical trials often echo findings from meta- analyses of several smaller studies.  Trends established by cumulative meta-analyses of previous studies are unlikely to be reversed.

60.  Model parameters are random variables, not unknown constants  Probability distribution quantifies this uncertainty  Prior before collecting data  Posterior after observing data  Likelihood is probability of observing data under model  Statements about probability of hypotheses and parameter values  Obtained by combining prior and likelihood with Bayes rule Bayesian Models

61.  Quantify knowledge of parameters  Incorporate all sources of variation in single model  Inference not restricted just to model parameters  Can be generalized to nonnormal priors and likelihoods (e.g. binomial)  Posterior densities need not be normally distributed like maximum likelihood estimates Advantages of Bayesian Models

62.  Infamous because random effects and fixed effects analysis lead to different conclusions –Random effects OR = 0.59 –Fixed effects OR = 1.02  Very large, influential clinical trial showed treatment gave no benefit  Contradicted earlier MA with large trial showing significant benefit Magnesium for AMI

63. Pooled Odds Ratio Mortality Observed Posterior StudyTreatedControlEst95% PIEst95% PIPr(OR<1) Morton1/402/360.440.0, 5.00.540.2, 1.60.89 Smith2/2007/2000.280.1, 1.50.460.1, 1.10.96 Abraham1/481/460.960.1, 15.80.610.2, 1.90.84 Feldstedt10/1508/1481.250.5, 3.30.860.4, 1.90.70 Rasmussen9/13523/1350.350.2, 0.80.430.2, 0.90.99 Ceremuz.1/253/230.280.0, 2.90.490.1, 1.40.92 Shechter I1/599/560.090.0, 0.70.380.1, 1.40.97 LIMIT 290/1159118/11570.740.6, 1.00.730.6, 1.00.99 ISIS-42216/290112103/290391.061.0, 1.11.061.0, 1.10.04 Shechter II2/8912/800.130.0, 0.60.360.1, 0.90.99 Singh6/7611/750.500.2, 1.40.540.2, 1.10.95 Pooled0.590.4, 0.90.550.3, 0.90.99 Meta-analysis for Magnesium Studies

64. Indirect Comparisons of Multiple Treatments Trial 1AB 2AB 3BC 4BC 5AC 6AC 7ABC We want to compare A vs. B Direct evidence from trials 1, 2 and 7 Indirect evidence from trials 3, 4, 5, 6 and 7

65. Randomized comparisons of antibiotics for acute sinusitis

66.  Regression analysis to identify correlations between treatment effects (outcomes) and covariates of interest (predictors)  Unit of analysis is the individual study Correlation implies treatment interaction Factors may be study-level or subject-level  Study-level factors include blinding, randomization, dosage, protocol  Subject-level factors include age, gender, race, blood pressure  i =  0 +   1 X i1 +   2 X i2 + … + u i Meta-Regression

67. Sources of Heterogeneity  Different study populations  Different treatments or protocols  Quality of technical design or execution  Random variation

68. Problems with Meta-Regression Number of studies usually small Data may be unavailable (not conceived or not reported) Covariates pre-selected (biased?) Little variation in range of mean predictor Subject-level factors can be affected by ecological bias Causality uncertain

69. Patient Level Analysis Principle: risk factors differ between patients Main model: multivariate regression of individual patient data Advantages  maximum information using patient as unit of analysis  direct interpretation for individual patient  no reporting bias  no ecologic bias Disadvantages  data difficult to obtain and frequently unavailable  retrieval bias  causality uncertain  costly

70. Meta-Regression vs. Individual Patient Regression Meta-Regression Individual Patient Regression Costcheapexpensive Factorsstudypatient and study Outcomesreportedupdated, complete Data Cleaningnot possiblereconciliation, missing data Biaspublicationdata retrieval, publication