Systematic Review: Meta-analysis II The nuts and bolts of the statistics Alka M. Kanaya, M.D. Assistant Professor of Medicine, Epi/Biostats April 19, 2007

Goals 1.Understand statistical issues for MA  summary estimate and variance models methods  heterogeneity  publication bias 2.Carry on an intelligent conversation with your statistician 3. Know if published MA used appropriate methods

Meta-analysis: the Steps 1.Formulate a question, eligibility criteria 2.Perform a systematic literature search 3.Abstract the data 4.Perform a statistical analysis 5.Calculate the summary effect size 6.Calculate the summary effect size for subgroups 7.Check for heterogeneity 8.Check for publication bias

Clinical Case  5 y.o. girl c/o ear pain and is found to have an acute otitis media.  Should she get antibiotics? Research Questions: 1.Are antibiotics effective for pain relief in children with acute OM? 2.Do antibiotics reduce complications of OM (mastoiditis, hearing problems)?

Systematic Review part  Inclusion criteria: –RCT of antibiotic vs. placebo –Children –Without tympanostomy tubes –With OM (regardless of setting of recruitment) –Patient-relevant outcomes  8 trials with total of 2,287 kids Glasziou, Cochrane Review, 2005

Goal #1: “Best” estimate  Combine findings from several studies to get the "best" estimate  Calculate weighted mean effect estimate, or a summary effect estimate 700Total RRNStudy Do antibiotics reduce pain? 3 RCTs: summary effect estimate= Σ (N i x effect estimate i ) = 640 =0.914 Σ(N i ) 700

Goal #1: Calculate weighted mean effect estimate Summary = Σ (weight i x effect estimate i ) = 30.3 = 0.99 effect estimate Σ(weight i ) 30.5 StudyNRRVar RRWeight Total700

Goal #2: Determine if the summary effect is significant  Calculate variance of summary effect estimate, or the 95% CI around the summary estimate Variance of summary estimate = 1 Σ(weights i ) Variance of summary estimate = _1_ = % CI = √0.03 = Summary OR and 95% CI = 0.99 ( )

Type of Model? Variance RR s = 1/  w i Weight i = 1 + D variance RR i Weight i = 1 variance RR i Variance of individual studies + variance of differences between studies Weights: variance of individual studies Existing studies are a random sample Existing studies are the entire population Goal: estimate the “true” effect Goal: weighted average of risk from existing studies Random EffectsFixed Effects

Model: Random Effects Model: Summary RR b Summary RR a b a

Random VS. Fixed Effects Models Practical Implications of the Choice  Summary estimates: usually similar  Variance: RE model produces large variance of the summary estimate  Confidence intervals: RE model produces wider confidence intervals  Statistical significance: less likely with RE model BOTTOM LINE:  If the individual study findings are similar, the model makes little difference in estimate or statistical significance.  If the individual study findings are heterogeneous, the model can affect the statistical significance.

Which method? ModelMethodEffect estimate Data Fixed Effects: Mantel- Haenszel Ratio (OR, RR)Crude (2x2) General Variance-based Ratio (OR, RR) Difference (risk, rate) Adjusted ratio & CI Crude (2x2) Random Effects: DerSimonian & Laird Ratio, difference Crude (2x2) General Variance-based Ratio (OR, RR) Difference (risk, rate) Adjusted ratio & CI Crude (2x2)

Which method to use? Types of Studies in Meta-analysis Method to Use Randomized trials: Any method (usually Mantel-Haenszel or DerSimonian & Laird Observational studies: General Variance Based

Mantel-Haenszel Method (Fixed Effects Model) DiseasedNot diseased Treated (exposed) a i c i Not treated (unexposed) b i d i OR i = a i / c i = a i x d i lnOR mh = Σ (w i x lnOR i ) b i / d i b i x c i Σw i variance lnOR i = variance OR mh = 1 a i b i c i d i Σ w i weight i = (w i ) = 1 variance lnOR i 95% CI = e lnORmh  (1.96 x √variance lnOR mh )

Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation Study 1 PerforationNo Perforation Antibiotic 1114 Placebo 3116 Study 2 PerforationNo Perforation Antibiotic 7 65 Placebo Calculate OR i for each study: OR 1 =1 x 116 = 0.34lnOR 1 = x 114 OR 2 = _______ = 0.58lnOR 2 = -0.54

Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation 2. Calculate variance lnOR i for each study: Var ln OR 1 = = Var ln OR 2 = ______________ = Calculate w i for each study: w 1 = 1 = w 2 = ________ = 3.85

Study 1 PerforationNo Perforation Antibiotic 1114 Placebo 3116 Study 2 PerforationNo Perforation Antibiotic 7 65 Placebo Calculate the w i x ln OR i for each study: w 1 x lnOR 1 = 0.74 x = w 2 x lnOR 2 = 3.85 x = Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation

5. Calculate the sum of the w i w 1 + w 2 = = Calculate lnOR mh = Σ (w i x lnOR i ) = = Σ w i 4.59 = OR mh = Calculate variance OR mh = 1 = 1 = 0.22 Σ w i Calculate 95% CI = e lnORmh + (1.96 x √ variance lnORmh) = e (1.96 x √ 0.22) = Summary OR = 0.53 (95% CI 0.21 – 1.34) Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation

Dersimonian and Laird Method (Random Effects Model) Similar formula to Mantel-Haenszel: ln OR dl = Σ (w i x ln OR i ) w i = 1 Σw i variance i + D  Where D gets larger as the OR (or effect estimate) of the individual studies vary from the summary estimate

General Variance-Based Method (Fixed or Random Effects) Confidence Intervals: ln OR s = Σ (w i x ln OR i )w i = 1 ______ Σw i variance lnOR i (+D) Variance lnOR i = ln OR i / OR l 2 or ln OR u / OR i Where OR i = OR on i th study OR l = lower bound of 95 % CI for i th study OR u = upper bound of 95 % CI for i th study  Should always be used for MA of Observational studies  Uses adjusted effect estimates  Preserves adjustment for confounding

Choice of Model and Method in Meta-Analysis What type of studies are you summarizing? Randomized TrialsObservational Studies  Either Model  Any Method  Either Model  Confidence Interval Method

Heterogeneity  Are you comparing apples and oranges?  Clinical heterogeneity: are studies asking same question?  Statistical heterogeneity: is the variation likely to have occurred by chance?  Measures how different each individual OR/RR is from the summary OR/RR.  Studies whose OR/RRs are very different from the summary OR/RRs contribute greatly to the heterogeneity, especially if they are weighted heavily.

Problem of Heterogeneity  Study findings are different and should not be combined StudyOR StudyOR

Statistical tests of Homogeneity  Is the variation in the individual study findings likely due to chance? H o : Effect estimate in each study is the same (or homogeneous) H a : Effect estimate in each study is not the same (or heterogeneous) Q = Σ(w i x (ln OR mh – ln OR i ) 2 ) df = (N studies -1) p < 0.05 or 0.10 = reject null, i.e., studies are heterogeneous  8 trials of antibiotics vs. P o for OM, pain outcome:  Q for homogeneity: p=0.91

Subgroup & Sensitivity Analysis  Subgroup Analysis – MA of a subgroup of eligible studies  age  ethnicity  risk factors  treatment  Sensitivity Analysis – add or delete questionable studies  eligibility  treatment

Subgroup Analysis OR95% CIN Ever user Of estrogen: All eligible studies Cohort studies Case-Control studies 2.3* 1.7* 2.4* Dose of estrogen: 0.3 mg mg  1.25 mg Duration of use: < 1 year 1-5 years 5-10 years  10 years * Regimen:Cyclic Daily 3.0* 2.9* * p for homogeneity < 0.05

Subgroup Analysis Antibiotics vs. Placebo for acute OM Outcome: abnormal tympanometry Study0 – 1 mo f/u RR (95% CI) 2 – 3 mo f/u RR (95% CI) Appelman, (0.25 – 1.26)NA Burke, (0.62 – 1.84)0.58 (0.31 – 1.08) Mygind, (0.49 – 1.94)1.09 (0.52 – 2.31) Summary estimate : 0.91 (0.62 – 1.32)0.75 (0.47 – 1.21) p-for-heterogeneity:

Sensitivity Analyses Performed to test the robustness of the findings To fairly assess and acknowledge the limitations Address publication bias (funnel plots, number needed to change result, etc..)

Sensitivity Analysis Aspirin + Heparin vs. Aspirin alone for Unstable Angina MI or Death Study (reference)AspirinAspirin + heparinRR (95% CI) Theroux et al, /121 (3%)2/122 (2%)0.50 ( ) RISC Group, /189 (4%)3/210 (1%)0.39 ( ) Cohen et al, /32 (3%)0/37 (0%)0.29 ( ) Cohen et al, /109 (8%)4/105 (4%)0.46 ( ) Holdright et al, /131 (31%)42/154 (27%)0.89 ( ) Gurfinkel et al, /73 (10%)4/70 (6%)0.60 ( ) Total/Summary:68/655 (10%)55/698 (8%)0.67 (0.44-1/02)  Remove Holdright: RR s = 0.45 (95% CI ) ; p-for-hetero=0.71  Add data from two additional trials of LMWH: RR s = 0.56 (95% CI ); p for heterogeneity: 0.52  Fixed effects model, Mantel-Haenszel method = same findings

Publication Bias  Published studies may not be representative of all studies ever conducted.  Selective publication of studies based on strength & direction of results & language.  AKA “positive outcome bias”

Minimizing Publication Bias Search bibliographies of published papers Consult with experts Search for unpublished data Clinical Trial Registries (NIH, VA) Institutional Review Boards Pharmaceutical companies Hand searches Consider studies not published in English Stern, BMJ, 2001

Statistical Approaches to Publication Bias  Correlation between study sample size (or weight or variance) and effect estimate  Funnel plot  Other fancy statistical methods: estimate number of unpublished studies that must exist to invalidate the results of the meta- analysis. “File drawer” “Fail-safe N” eliminate the studies that may have been published due to bias

Association of Estrogen use and Endometrial Cancer Correlation of sample size and RR: rho = 0.68; p = 0.08 FUNNEL GRAPH Relative Risk of Endometrial Cancer

RCTs of Heparin plus ASA vs. ASA Correlation of sample size and RR: rho = 0.25; p= 0.64 FUNNEL GRAPH Relative Risk for MI or Death Favors Heparin+ASA Favors ASA Sample Size

Presentation of the Results Tables:  Study Characteristics  population  sample size  definition of intervention  definition of the outcome  important design features (validity of the data) - randomization - blinding - follow-up - compliance  Study Findings  main and secondary outcomes  outcomes by subgroup  sensitivity analysis findings

Table 1. Characteristics of 6 randomized trials of aspirin + heparin vs. aspirin alone to prevent MI and death in patients admitted with unstable angina Study (ref.)BlindingAspirin DoseGoal PTTDuration of Heparin Theroux, Participants & Investigators 325 mg twice per day x normal6 days RISC Group, None75 mg dailyNot stated5 days Cohen, None 80/325 mg daily* 2 x normal3-4 days Cohen, Participants162.5 mg daily2 x normal3-4 days Holdright, Participants150 mg daily1.5-2 x normal2 days Gurfinkel, Participants & Investigators 200 mg daily2 x normal5-7 days

Bent, AMJ, 1999 No Caption Found Antibiotics vs. Placebo in Acute Bronchitis

Take Home Messages  You can do a meta-analysis  Good start on becoming an expert in your field  Your work should be reproducible  Your conclusions should be obvious  Include a statistician on you team