1. Systematic Review: Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF

2. 8 Steps to Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

4. Why conduct a systematic review?  The best way to summarize evidence on a scientific topic  Concisely communicates findings to others in the field  Identifies author(s) as experts  Identifies areas for future study  Perfect for background of grants  Don’t need to do primary data collection (so can be done while waiting for data from other projects)  You have to do the work anyway, so might as well get a publication!  You can effect change in clinical management

5. Cumulative Meta-analysis Antman EM et al: JAMA. 1992;268:240-248

6. Systematic Review: Clinical Implications (Antiarrhythmic Drugs for Acute MI) Teo KK et al. JAMA. 1993;270:1589-1595

7. Sample Systematic Reviews  Kangelaris KN, Bent S, Nussbaum RL, Garcia DA, Tice JA. Genetic testing before anticoagulation? A systematic review of the safety and efficacy of pharmacogenetic dosing of warfarin. Journal of General Internal Medicine (in press).  Nguyen SP, Bent S, Chen Y, Terdiman JP. Gender as a Risk Factor for Advanced Neoplasia and Colorectal Cancer: A Systematic Review and Meta-analysis. Clinical Gastroenterology (in press).  Simon J, Chen Y, Bent S. The relation of alpha-linoleic acid to the risk of prostate cancer: a systematic review. Am J Clin Nutr. 2009;89:1-7S.  Li J, Winston LG, Moore DH, Bent S. Efficacy of short-course antibiotic regimens for community-acquired pneumonia: a meta- analysis. American Journal of Medicine. 2007;120(9):783-90.  Margaretten M, Kohlwes J, Moore D, Bent S. The rational clinical examination: does this patient have septic arthritis. JAMA. 2007;297:1478-1488.

8. 8 Steps of Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

9. Create a Summary Measure  Before we get to the mechanics of a summary measure….  Be sure to provide your audience with a concise summary table  A “visual meta-analysis”  Readers should be able to examine Table 1 and reach their own conclusions about the data

10. Example  Antibiotics for acute bronchitis.  After search and application of inclusion/exclusion criteria, 8 studies were included.

11. RCTs in Acute Bronchitis Study, yrNAbxOutcome Result * Stott, 76207DoxyDays of Yellow Spit0.6 (-0.2 to 1.4) Franks, 8454TMP/SCough Amount Score0.2 (-0.2 to 0.6) Williamson, 8469DoxyDays of Purulent Sputum-0.2 (-1.2 to 0.8) Dunlay, 8745ErythroSputum production score0.5 (0.1 to 0.9) Scherl, 8731DoxyDays of sputum1.9 (-0.2 to 4.0) Verheij, 94140DoxyDays of productive cough0.5 (-0.4 to 1.4) Hueston, 9423ErythroDays of productive cough-0.4 (-2.4 to 1.6) King, 9691ErythroDays of sputum production0.7 (-1.3 to 2.7) * Positive numbers indicate antibiotics are superior to placebo

13. How do you create a summary measure?  Clinical example: 5 year old girl presents with ear pain and is found to have an acute otitis media.  Should she get antibiotics? Research Questions: 1.In children with OM, are antibiotics effective for pain relief? 2.In children with OM, do antibiotics reduce the rate of complications (mastoiditis, hearing problems)?

14. 3 studies are identified (examining effect of Abx on Pain)  Study 1: N = 100RR=1.41  Study 2: N=200RR=0.98  Study 3: N=300RR=1.01  You could take the average effect: (1.41 + 0.98 + 1.01) / 3 = 1.13  Is this a good summary measure?

15. Summary measure weighted by sample size  Provide “weight” for studies based on their sample size 600Total 1.013003 0.982002 1.411001 RRNStudy summary effect estimate= Σ (N i x effect estimate i ) = 640 =1.07 Σ(N i ) 600

16. More refined: Provide “weight” by using inverse of variance Summary = Σ (weight i x effect estimate i ) = 30.5 = 1.00 effect estimate Σ(weight i ) 30.3 StudyNRRVar RRWeight 11001.413.00.33 22000.980.110 33001.010.0520 Total700

17. Does the largest study always have the lowest variance and therefore the greatest “weight”?  Dichotomous outcomes  Continuous outcomes

18. Confidence Intervals Around Summary Effect  Calculate variance of summary effect estimate, or the 95% CI around the summary estimate Variance of summary estimate = 1 Σ(weights i ) Variance of summary estimate = _1_ =.03 30.5 95% CI = + 1.96 √0.03 = + 0.34 Summary OR and 95% CI = 1.00 (0.65 - 1.33)

19. Type of Model? Variance RR s = 1/  w i Weight i = 1 variance RR i + D Weight i = 1 variance RR i Variance of individual studies + variance of differences between studies Weights: variance of individual studies Existing studies are a random sample Existing studies are the entire population Goal: estimate the “true” effect Goal: weighted average of risk from existing studies Random EffectsFixed Effects

20. Formulas for D

21. Fixed Effects Model: Random Effects Model: Summary RR b Summary RR a b a

22. Random VS. Fixed Effects Model Practical Implications of the Choice  Summary estimates: usually similar  Variance: RE model produces larger variance of the summary estimate  Confidence intervals: RE model produces wider confidence intervals  Statistical significance: less likely with RE model BOTTOM LINE:  If the individual study findings are similar, the model makes little difference in estimate or statistical significance.  If the individual study findings are heterogeneous, the model can affect the statistical significance.

23. Which method for creating summary measure? ModelMethodEffect estimate Data Fixed Effects: Mantel- Haenszel Ratio (OR, RR)Crude (2x2) General Variance-based Ratio (OR, RR) Difference (risk, rate) Adjusted ratio & CI Crude (2x2) Random Effects: DerSimonian & Laird Ratio, difference Crude (2x2) General Variance-based Ratio (OR, RR) Difference (risk, rate) Adjusted ratio & CI Crude (2x2)

24. Mantel-Haenszel Method (Fixed Effects Model) DiseasedNot diseased Treated (exposed) a i c i Not treated (unexposed) b i d i OR i = a i / c i = a i x d i lnOR mh = Σ (w i x lnOR i ) b i / d i b i x c i Σw i variance lnOR i = 1 + 1 + 1+ 1 variance OR mh = 1 a i b i c i d i Σ w i weight i = (w i ) = 1 variance lnOR i 95% CI = e lnORmh  (1.96 x √variance lnOR mh )

25. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation Study 1 PerforationNo Perforation Antibiotic 1114 Placebo 3116 Study 2 PerforationNo Perforation Antibiotic 7 65 Placebo 12 65 1. Calculate OR and lnOR for each study: OR 1 =1 x 116 = 0.34lnOR 1 = -1.08 3 x 114 OR 2 = 7 x 65 = 0.58lnOR 2 = -0.54 12 x 65

26. Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation 2. Calculate variance lnOR i for each study: Var ln OR 1 = 1 + 1 + 1 + 1 = 1.35 1 3 114 116 Var ln OR 2 = 1 + 1 + 1 + 1 = 0.26 7 12 65 65 3. Calculate w i for each study: w 1 = 1 = 0.74 1.35 w 2 = 1 = 3.85 0.26

27. Study 1 PerforationNo Perforation Antibiotic 1114 Placebo 3116 Study 2 PerforationNo Perforation Antibiotic 7 65 Placebo 12 65 4.Calculate the w i x ln OR i for each study: w 1 x lnOR 1 = 0.74 x -1.08 = -0.80 w 2 x lnOR 2 = 3.85 x -0.54 = -2.08 Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation

28. 5. Calculate the sum of the w i w 1 + w 2 = 0.74 + 3.85 = 4.59 6.Calculate lnOR mh = Σ (w i x lnOR i ) = -0.80 + -2.08 = -0.63 Σ w i 4.59 = OR mh = 0.53 7.Calculate variance OR mh = 1 = 1 = 0.22 Σ w i 4.59 8.Calculate 95% CI = e lnORmh + (1.96 x √ variance lnORmh) = e -.63 + (1.96 x √ 0.22) = 0.21 - 1.34 Summary OR = 0.53 (95% CI 0.21 – 1.34) Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation

29. Dersimonian and Laird Method (Random Effects Model) Similar formula to Mantel-Haenszel: ln OR dl = Σ (w i x ln OR i ) w i = 1 Σw i variance i + D  Where D gets larger as the OR (or effect estimate) of the individual studies vary from the summary estimate

30. But…All you need to know is:  When combined, individual study effect estimates are weighted by their inverse variance  Variance is related to sample size AND # of events (dichotomous) and precision (continuous)  Fixed effects just combines all weighted estimates, while random effects “penalizes” estimates for variation between studies

31. 8 Steps to Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

32. Heterogeneity  Are you comparing apples and oranges?  Clinical heterogeneity: are studies asking same question?  Statistical heterogeneity: is the variation likely to have occurred by chance?  Measures how far each individual OR/RR is from the summary OR/RR.  Studies whose OR/RRs are very different from the summary OR/RRs contribute greatly to the heterogeneity, especially if they are weighted heavily.

33. Heterogeneity  Refers to the degree that the study results differ  Visual Approach  Statistical Approach Q =sum [weight i x (ES s – ES i )] p < 0.05 indicates heterogeneity

34. Summary RR = 0.93 (0.87-0.99)

35. Problem of Heterogeneity  Study findings are different – should they be combined? StudyOR 10.01 21.0 310.0 StudyOR 10.35 20.56 30.97 41.15 51.75 61.95

36. Statistical tests of Heterogeneity  Is the variation in the individual study findings likely due to chance? H o : Effect estimate in each study is the same (or homogeneous) H a : Effect estimate in each study is not the same (or heterogeneous) Q = Σ(w i x (ln OR mh – ln OR i ) 2 ) df = (N studies -1) p < 0.05 or 0.10 = reject null, i.e., studies are heterogeneous  8 trials of antibiotics vs. P o for OM, pain outcome:  Q for heterogeneity: p=0.91

37. Heterogeneity – Interpret Findings (Example: RR of Colon CA, Men vs. Women)

38. 8 Steps to Sytematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

39. Assessing for Publication Bias  Publication Bias – the publication or “non-publication” of research findings, depending on the nature and direction of the results.  Rosenthal, 1979 – published an article describing the “file-drawer problem” that journals publish only 5% of all negative studies, while the file drawers in the back of the lab contain the other 95%.

40. Methods for Assessing Publication Bias  Funnel plots – simple scatter plots of treatment effects (horizontal axis) vs. some measure of study size (vertical axis).  Choice of axes –Log scale for treatment effects (to ensure that treatment effects in opposite directions are the same distance from 1.0 – e.g., 0.5 and 2.0) –Standard error for measure of sample size Power depends on both sample size and # events Standard error is consistent with the statistical tests

42. Funnel Plot of Log Relative Risk vs Standard Error Log Relative Risk Standard error 5 4 3 2 1 0

43. Example: ALA and Prostate Cancer Risk RR=1.2 (1.01 to 1.43), Test for heterogeneity, p=0.00

44. ALA – Funnel Plot

45. Funnel Plot with Imputed Values for Publication Bias RR=0.94, 95% CI: 0.79-1.17

46. Publication bias caveat  Funnel plot asymmetry does not always indicate bias –It is possible that smaller studies enrolled higher risk patients, for example, and therefore found a greater effect. –Small studies are often conducted before larger studies. In the intervening years, other interventions may have improved, thus reducing the relative efficacy of the treatment.

47. Statistical methods to assess publication bias  Examine associations between study size and treatment effect. –Sensitivity is poor when < 20 studies  Begg’s test: an adjusted rank correlation  Egger’s test: a weighted regression of effect size vs. standard error. –Basically asks if the regression line has a non-zero slope –More sensitive than Begg’s test, but more false positives, especially when 1) large treatment effects, 2) few events per trial, 3) all trials of similar size. (In these cases, one may decide a priori to use Begg’s test).

49. Begg's Test adj. Kendall's Score (P-Q) = -30 Std. Dev. of Score = 14.58 Number of Studies = 12 z = -2.06 Pr > |z| = 0.040 z = 1.99 (continuity corrected) Pr > |z| = 0.047 (continuity corrected) Egger's test ------------------------------------------------------------------------------ Std_Eff | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- slope |.9810716.1103858 8.89 0.000.7351168 1.227026 bias | -.9911295.3236382 -3.06 0.012 -1.71224 -.2700187 ------------------------------------------------------------------------------

50. 8 Steps to Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

51. Subgroup & Sensitivity Analysis  Subgroup Analysis – MA of a subgroup of eligible studies  age  ethnicity  risk factors  treatment  Sensitivity Analysis – add or delete questionable studies  eligibility  treatment

52. Subgroup Analysis OR95% CIN Ever user Of estrogen: All eligible studies Cohort studies Case-Control studies 2.3* 1.7* 2.4* 2.1 - 2.5 1.3 - 2.1 2.2 - 2.6 29 4 25 Dose of estrogen: 0.3 mg 0.625 mg  1.25 mg 3.9 3.4 5.8 1.6 - 9.5 2.0 - 5.6 4.5 - 7.5 349349 Duration of use: < 1 year 1-5 years 5-10 years  10 years 1.4 2.8 5.9 9.5* 1.0 - 1.8 2.3 - 3.5 4.7 - 7.5 7.4 - 12.3 9 12 10 Regimen:Cyclic Daily 3.0* 2.9* 2.4 - 3.8 2.2 - 3.8 8888 * p for heterogeneity < 0.05

54. Analytical Methods: Summary Points  Always start the meta-analysis with a “visual meta-analysis” (i.e., a great table 1). –A clinician should be able to interpret the results  Step 1: Calculate a summary measure = “weighted mean effect estimate” –You can combine anything, but use judgment  Step 2: Assess for heterogeneity –Heterogeneity is not always a problem  Step 3: Assess for publication bias –Both visual and statistical methods  Step 4: Perform subgroup/sensitivity analyses –Ideally specify these a priori

55. 8 Steps to Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation