Systematic Review: Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF.

Systematic Review: Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF

8 Steps to Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

Juni et al, Hazards of scoring the quality of clinical trials. JAMA. 1999;282:1054-60.

Why conduct a systematic review?  The best way to summarize evidence on a scientific topic  Concisely communicates findings to others in the field  Identifies author(s) as experts  Identifies areas for future study  Perfect for background of grants  Don’t need to do primary data collection (so can be done while waiting for data from other projects)  You have to do the work anyway, so might as well get a publication!  You can effect change in clinical management

Cumulative Meta-analysis Antman EM et al: JAMA. 1992;268:240-248

Systematic Review: Clinical Implications (Antiarrhythmic Drugs for Acute MI) Teo KK et al. JAMA. 1993;270:1589-1595

Sample Systematic Reviews  Kangelaris KN, Bent S, Nussbaum RL, Garcia DA, Tice JA. Genetic testing before anticoagulation? A systematic review of the safety and efficacy of pharmacogenetic dosing of warfarin. Journal of General Internal Medicine (in press).  Nguyen SP, Bent S, Chen Y, Terdiman JP. Gender as a Risk Factor for Advanced Neoplasia and Colorectal Cancer: A Systematic Review and Meta-analysis. Clinical Gastroenterology. 2009;7:676-81.  Simon J, Chen Y, Bent S. The relation of alpha-linoleic acid to the risk of prostate cancer: a systematic review. Am J Clin Nutr. 2009;89:1-7S.  Li J, Winston LG, Moore DH, Bent S. Efficacy of short-course antibiotic regimens for community-acquired pneumonia: a meta- analysis. American Journal of Medicine. 2007;120(9):783-90.  Margaretten M, Kohlwes J, Moore D, Bent S. The rational clinical examination: does this patient have septic arthritis. JAMA. 2007;297:1478-1488.

Sample Systematic Reviews Hsu J, Kohlwes J, Bent S. Efficacy of antifungal therapy in chronic rhinosinusitis: A systematic review. J Allergy Clin Immunol. 2010 125:2 Guarnieri M, Bent S. Death from coronary artery disease in patients with systemic lupus erythematosus: a systematic review and meta-analysis of mortality cohort studies. (submitted to Arthritis Care and Research 1/2012). Lee JK, Liles EG, Bent S, Levin TR, Corley DA. Diagnostic Accuracy of Fecal Immunochemical Tests for Colorectal Cancer: Systematic Review and Meta-analysis (submitted to JAMA 4/2013).

8 Steps of Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

Create a Summary Measure  Before we get to the mechanics of a summary measure….  Be sure to provide your audience with a concise summary table  A “visual meta-analysis”  Readers should be able to examine Table 1 and reach their own conclusions about the data

Example  Antibiotics for acute bronchitis.  After search and application of inclusion/exclusion criteria, 8 studies were included.

RCTs in Acute Bronchitis Study, yrNAbxOutcome Result * Stott, 76207DoxyDays of Yellow Spit0.6 (-0.2 to 1.4) Franks, 8454TMP/SCough Amount Score0.2 (-0.2 to 0.6) Williamson, 8469DoxyDays of Purulent Sputum-0.2 (-1.2 to 0.8) Dunlay, 8745ErythroSputum production score0.5 (0.1 to 0.9) Scherl, 8731DoxyDays of sputum1.9 (-0.2 to 4.0) Verheij, 94140DoxyDays of productive cough0.5 (-0.4 to 1.4) Hueston, 9423ErythroDays of productive cough-0.4 (-2.4 to 1.6) King, 9691ErythroDays of sputum production0.7 (-1.3 to 2.7) * Positive numbers indicate antibiotics are superior to placebo

How do you create a summary measure?  Clinical example: 5 year old girl presents with ear pain and is found to have an acute otitis media.  Should she get antibiotics? Research Questions: 1.In children with OM, are antibiotics effective for pain relief? 2.In children with OM, do antibiotics reduce the rate of complications (mastoiditis, hearing problems)?

3 studies are identified (examining effect of Abx on Pain)  Study 1: N = 100RR=1.41  Study 2: N=200RR=0.98  Study 3: N=300RR=1.01  You could take the average effect: (1.41 + 0.98 + 1.01) / 3 = 1.13  Is this a good summary measure?

Summary measure weighted by sample size  Provide “weight” for studies based on their sample size 600Total 1.013003 0.982002 1.411001 RRNStudy summary effect estimate= Σ (N i x effect estimate i ) = 640 =1.07 Σ(N i ) 600

More refined: Provide “weight” by using inverse of variance Summary = Σ (weight i x effect estimate i ) = 30.5 = 1.00 effect estimate Σ(weight i ) 30.3 StudyNRRVar RRWeight 11001.413.00.33 22000.980.110 33001.010.0520 Total700

Does the largest study always have the lowest variance and therefore the greatest “weight”?  Dichotomous outcomes  Continuous outcomes

Confidence Intervals Around Summary Effect  Calculate variance of summary effect estimate, or the 95% CI around the summary estimate Variance of summary estimate = 1 Σ(weights i ) Variance of summary estimate = _1_ =.03 30.5 95% CI = + 1.96 √0.03 = + 0.34 Summary OR and 95% CI = 1.00 (0.65 - 1.33)

Type of Model? Variance RR s = 1/  w i Weight i = 1 variance RR i + D Weight i = 1 variance RR i Variance of individual studies + variance of differences between studies Weights: variance of individual studies Existing studies are a random sample Existing studies are the entire population Goal: estimate the “true” effect Goal: weighted average of risk from existing studies Random EffectsFixed Effects

Formulas for D

Fixed Effects Model: Random Effects Model: Summary RR b Summary RR a b a

Random VS. Fixed Effects Model Practical Implications of the Choice  Confidence intervals: RE model produces wider confidence intervals  Statistical significance: less likely with RE model BOTTOM LINE:  If the individual study findings are similar, the model makes little difference in estimate or statistical significance.  If the individual study findings are heterogeneous, the model can affect the statistical significance.

Mantel-Haenszel Method (Fixed Effects Model) DiseasedNot diseased Treated (exposed) a i c i Not treated (unexposed) b i d i OR i = a i / c i = a i x d i lnOR mh = Σ (w i x lnOR i ) b i / d i b i x c i Σw i variance lnOR i = 1 + 1 + 1+ 1 variance OR mh = 1 a i b i c i d i Σ w i weight i = (w i ) = 1 variance lnOR i 95% CI = e lnORmh  (1.96 x √variance lnOR mh )

Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation Study 1 PerforationNo Perforation Antibiotic 1114 Placebo 3116 Study 2 PerforationNo Perforation Antibiotic 7 65 Placebo 12 65 1. Calculate OR and lnOR for each study: OR 1 =1 x 116 = 0.34lnOR 1 = -1.08 3 x 114 OR 2 = 7 x 65 = 0.58lnOR 2 = -0.54 12 x 65

Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation 2. Calculate variance lnOR i for each study: Var ln OR 1 = 1 + 1 + 1 + 1 = 1.35 1 3 114 116 Var ln OR 2 = 1 + 1 + 1 + 1 = 0.26 7 12 65 65 3. Calculate w i for each study: w 1 = 1 = 0.74 1.35 w 2 = 1 = 3.85 0.26

Study 1 PerforationNo Perforation Antibiotic 1114 Placebo 3116 Study 2 PerforationNo Perforation Antibiotic 7 65 Placebo 12 65 4.Calculate the w i x ln OR i for each study: w 1 x lnOR 1 = 0.74 x -1.08 = -0.80 w 2 x lnOR 2 = 3.85 x -0.54 = -2.08 Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation

5. Calculate the sum of the w i w 1 + w 2 = 0.74 + 3.85 = 4.59 6.Summary lnOR mh = Σ (w i x lnOR i ) = -0.80 + -2.08 = -0.63 Σ w i 4.59 = OR mh = 0.53 7.Calculate variance OR mh = 1 = 1 = 0.22 Σ w i 4.59 8.Calculate 95% CI = e lnORmh + (1.96 x √ variance lnORmh) = e -.63 + (1.96 x √ 0.22) = 0.21 - 1.34 Summary OR = 0.53 (95% CI 0.21 – 1.34) Randomized Trials of Antibiotic Rx for acute OM to prevent TM perforation

Dersimonian and Laird Method (Random Effects Model) Similar formula to Mantel-Haenszel: ln OR dl = Σ (w i x ln OR i ) w i = 1 Σw i variance i + D  Where D gets larger as the OR (or effect estimate) of the individual studies vary from the summary estimate

But…All you need to know is:  When combined, individual study effect estimates are weighted by their inverse variance  Variance is related to sample size AND # of events (dichotomous) and precision (continuous)  Fixed effects just combines all weighted estimates, while random effects “penalizes” estimates for variation between studies

Heterogeneity  Are you comparing apples and oranges?  Clinical heterogeneity: are studies asking same question?  Statistical heterogeneity: is the variation likely to have occurred by chance?  Measures how far each individual OR/RR is from the summary OR/RR.  Studies whose OR/RRs are very different from the summary OR/RRs contribute greatly to the heterogeneity, especially if they are weighted heavily.

Heterogeneity  Refers to the degree that the study results differ  Visual Approach  Statistical Approach Q =sum [weight i x (ES s – ES i )] p < 0.05 indicates heterogeneity

Summary RR = 0.93 (0.87-0.99)

Problem of Heterogeneity  Study findings are different – should they be combined? StudyOR 10.01 21.0 310.0 StudyOR 10.35 20.56 30.97 41.15 51.75 61.95

Statistical tests of Heterogeneity  Is the variation in the individual study findings likely due to chance? H o : Effect estimate in each study is the same (or homogeneous) H a : Effect estimate in each study is not the same (or heterogeneous) Q = Σ(w i x (ln OR mh – ln OR i ) 2 ) df = (N studies -1) p < 0.05 or 0.10 = reject null, i.e., studies are heterogeneous

Heterogeneity – Interpret Findings (Example: RR of Colon CA, Men vs. Women)

8 Steps to Sytematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

Assessing for Publication Bias  Publication Bias – the publication or “non-publication” of research findings, depending on the nature and direction of the results.  Rosenthal, 1979 – published an article describing the “file-drawer problem” that journals publish only 5% of all negative studies, while the file drawers in the back of the lab contain the other 95%.

Methods for Assessing Publication Bias  Funnel plots – simple scatter plots of treatment effects (horizontal axis) vs. some measure of study size (vertical axis).  Choice of axes –Log scale for treatment effects (to ensure that treatment effects in opposite directions are the same distance from 1.0 – e.g., 0.5 and 2.0) –Standard error for measure of sample size Power depends on both sample size and # events Standard error is consistent with the statistical tests

Funnel Plot of Log Relative Risk vs Standard Error Log Relative Risk Standard error 5 4 3 2 1 0

Example: ALA and Prostate Cancer Risk RR=1.2 (1.01 to 1.43), Test for heterogeneity, p=0.00

ALA – Funnel Plot

Funnel Plot with Imputed Values for Publication Bias RR=0.94, 95% CI: 0.79-1.17

Publication bias caveat  Funnel plot asymmetry does not always indicate bias –It is possible that smaller studies enrolled higher risk patients, for example, and therefore found a greater effect. –Small studies are often conducted before larger studies. In the intervening years, other interventions may have improved, thus reducing the relative efficacy of the treatment.

Statistical methods to assess publication bias  Examine associations between study size and treatment effect. –Sensitivity is poor when < 20 studies  Begg’s test: an adjusted rank correlation  Egger’s test: a weighted regression of effect size vs. standard error. –Basically asks if the regression line has a non-zero slope –More sensitive than Begg’s test, but more false positives, especially when 1) large treatment effects, 2) few events per trial, 3) all trials of similar size. (In these cases, one may decide a priori to use Begg’s test).

Begg's Test adj. Kendall's Score (P-Q) = -30 Std. Dev. of Score = 14.58 Number of Studies = 12 z = -2.06 Pr > |z| = 0.040 z = 1.99 (continuity corrected) Pr > |z| = 0.047 (continuity corrected) Egger's test ------------------------------------------------------------------------------ Std_Eff | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- slope |.9810716.1103858 8.89 0.000.7351168 1.227026 bias | -.9911295.3236382 -3.06 0.012 -1.71224 -.2700187 ------------------------------------------------------------------------------

Subgroup & Sensitivity Analysis  Subgroup Analysis – MA of a subgroup of eligible studies  age  ethnicity  risk factors  treatment  Sensitivity Analysis – add or delete questionable studies  eligibility  treatment

Subgroup Analysis OR95% CIN Ever user Of estrogen: All eligible studies Cohort studies Case-Control studies 2.3* 1.7* 2.4* 2.1 - 2.5 1.3 - 2.1 2.2 - 2.6 29 4 25 Dose of estrogen: 0.3 mg 0.625 mg  1.25 mg 3.9 3.4 5.8 1.6 - 9.5 2.0 - 5.6 4.5 - 7.5 349349 Duration of use: < 1 year 1-5 years 5-10 years  10 years 1.4 2.8 5.9 9.5* 1.0 - 1.8 2.3 - 3.5 4.7 - 7.5 7.4 - 12.3 9 12 10 Regimen:Cyclic Daily 3.0* 2.9* 2.4 - 3.8 2.2 - 3.8 8888 * p for heterogeneity < 0.05

Analytical Methods: Summary Points  Always start the meta-analysis with a “visual meta-analysis” (i.e., a great table 1). –A clinician should be able to interpret the results  Step 1: Calculate a summary measure = “weighted mean effect estimate” –You can combine anything, but use judgment  Step 2: Assess for heterogeneity –Heterogeneity is not always a problem  Step 3: Assess for publication bias –Both visual and statistical methods  Step 4: Perform subgroup/sensitivity analyses –Ideally specify these a priori

Can you conduct a systematic review when there are only a few studies?

Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials  Objectives To determine whether parachutes are effective in preventing major trauma related to gravitational challenge.  Design Systematic review of randomised controlled trials.  Data sources: Medline,Web of Science, Embase, and the Cochrane Library databases; appropriate internet sites and citation lists.  Study selection: Studies showing the effects of using a parachute during free fall.  Main outcome measure Death or major trauma.  Results We were unable to identify any randomised controlled trials of parachute intervention.  Conclusions As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute. Smith GCS and Pill JP. BMJ 2003;327:1459–61

Advanced Topics  Individual participant data  Missing data  Different types of data  Observational studies  Generalized synthesis of evidence  Meta-regression  Critique of a systematic review

Different types of data  Different scales (example)  Ordinal data  Binary data  Continuous outcomes  Diagnostic tests (example)

RCTs in Acute Bronchitis: Different Scales Study, yr NAbxOutcomeResult Stott, 76 207Doxy Days of Yellow Spit 0.6 (-0.2 to 1.4) Franks, 84 54TMP/S Cough Amount Score 0.2 (-0.2 to 0.6) Williamson, 84 69Doxy Days of Purulent Sputum -0.2 (-1.2 to 0.8) Dunlay, 87 45Erythro Sputum production score 0.5 (0.1 to 0.9) Scherl, 87 31Doxy Days of sputum 1.9 (-0.2 to 4.0) Verheij, 94 140Doxy Days of productive cough 0.5 (-0.4 to 1.4) Hueston, 94 23Erythro Days of productive cough -0.4 (-2.4 to 1.6) King, 96 91Erythro Days of sputum production 0.7 (-1.3 to 2.7)

Problem  How do you combine studies with slightly different outcomes?  Option 1: - don’t do it  Option 2: Transform all outcomes to an effect size

What is an Effect Size?  Effect size – a way of expressing results in a common metric  Units – standard deviation

Effect Size  ES = X 1 – X 2 SD pooled 1.ES increases as difference between means increases 2.ES increases as SD decreases 3.ES is expressed in units of SD 4.Summary ES combines the weighted ES from each study.

Effect Size

 Rough Estimates –SMALL0.2 –MEDIUM0.5 –LARGE>0.7  Context –Mean Duration of Cough = 8 days –Standard Deviation = 3 days

Main Result  Summary ES = 0.21 (95% CI 0.05 to 0.36)

Summary Mean Differences Outcome Measure Summary Mean Difference (95% CI) Days of Productive Cough (6 studies) 0.4 days (-0.1 to 0.8) Days of cough (4 studies) 0.5 days (-0.1 to 1.1) Time off work (6 studies) 0.3 days (-0.6 to 1.1)

Different Types of Data: Diagnostic Tests

Sensitivity and Specificity Sensitivity TP/(TP + FN) Positive in Disease Specificity TN/(TN + FP) Negative in Health TNFNTest - FPTPTest + Disease - Disease +

(+) Likelihood Ratio = Sensitivity 1-Specificity (-) Likelihood Ratio = 1-Sensitivity Specificity

Does this patient have a specific disease?  What we thought before (pre-test probability)  + Clinical information (diagnostic test, LR)  = What we think after (post-test probability)

Diagnostic OR = +LR/-LR = TP x TN / FP x FN

Example: US and CT for Appendicitis  Goal: to determine whether US or CT is a “better” test for the evaluation of suspected appendicitis.  Diagnostic tests are complicated because there are 5 potential outcomes to summarize –LR+, LR- –Sensitivity, Specificity –Diagnostic OR –Assess heterogeneity, publication bias for EACH outcome

Advanced Topics  Individual participant data  Missing data  Different types of data  Observational studies  Generalized synthesis of evidence  Meta-regression  Critique of a systematic review

Meta-regression  Examines whether the study effects (outcomes) are related to one or more of the study characteristics.  Can be used to understand/explain heterogeneity.  Can be thought of as an epidemiological study of the trials or studies.

Clinical Questions: Meta-Regression Are there certain situations in which BCG may be more effective for preventing TB?

Meta-regression: example StudyOR 95% CI 10.391 0.121, 1.262 20.189 0.077, 0.462 30.250 0.069, 0.909 40.233 0.176, 0.308 50.803 0.514, 1.256 60.384 0.316, 0.466 70.195 0.077, 0.497 81.012 0.894, 1.146 90.624 0.391, 0.996 100.246 0.144, 0.422 110.711 0.571, 0.886 121.563 0.373, 6.548 130.983 0.582, 1.661  BCG vaccine: used to prevent tuberculosis  Odds ratio estimates from 13 trials (right)  Scientists have suggested that effects may be related to geographic latitude

Funnel Plot

Funnel Plot – Organized by Latitude

Meta-regression: example, continued Log odds ratio versus absolute latitude :

Meta-regression: example, cont Same plot, showing precision:

Meta-regression: example, cont Same plot, with fitted (meta-)regression line:

Meta-regression: example, cont  Is the slope of the line significantly different from 0?  If yes, we conclude that the study effects are in fact related to latitude

Meta-regression: details  In a regression model for the data: each study represents one observation  Weights equal to the study precision  Random effects model (recommended)  Built-in function in Stata: ‘metareg’

Critique of a Systematic Review  1. Research Question  2. Protocol  3. Search  4. Study selection (inclusion/exclusion)  5. Quality assessment  6. Data abstraction  7. Analysis –A) Create summary measure –B) Assess for heterogeneity –C) Assess for publication bias –D) Conduct sensitivity/subgroup analyses –E) Advanced issues/techniques  8. Interpretation

Reviewing Journal Articles  Very little formal teaching  “Because reviews are often highly negative, the new researcher implicitly learns from the negative reviews received on his or her own submitted papers that reviews are supposed to be negative. It is as if the implicit message is: A reviewer’s job is to criticize the manuscript.”

12 Tips on Reviewing Articles  1. Know your mission  2. Be speedy  3. Read carefully  4. Say positive things in your review  5. Don’t exhibit hostility  6. Keep it brief  7. Don’t nitpick  8. Develop your own style  9. Be careful in recommending further experiments  10. Watch for egocentrism  11. Make a recommendation  12. Sign your review http://www.psychologicalscience.org/observer/getArticle.cfm?id=2157

Conclusions  You can combine almost anything  Use clinical judgment to guide you in deciding how and whether to combine studies.  Remember the main mission of a systematic review: to summarize a body of literature in a concise and clear way.  Get statistical input as needed.

Systematic Review: Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF.

Similar presentations

Presentation on theme: "Systematic Review: Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Systematic Review: Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF.

Similar presentations

Presentation on theme: "Systematic Review: Analytical Methods of Meta-analysis Stephen Bent, M.D. Assistant Professor of Medicine, Epidemiology and Biostatistics UCSF."— Presentation transcript:

Similar presentations

About project

Feedback