Systematic Review Systematic review Why do we worry about reviews with misleading results? How does a systematic review protect against misleading results Understanding inconsistency in systematic reviews

Thrombolytic Therapy Textbook/Review Recommendations Cumulative 0.5 1.0 2.0 Year RCTs Pts 1 23 1960 Experimental Not Mentioned Rare/Never Routine Specific 2 65 1965 3 149 21 4 316 5 1970 7 1793 1 10 10 2544 1 2 11 2651 P<.01 15 3311 2 8 17 3929 22 5452 7 23 5767 1980 8 1 27 6125 12 P<.001 30 6346 M 8 1985 1 4 33 6571 M 7 43 21 059 1 3 M 54 22 051 P<.00001 5 2 2 65 47 185 M 1 67 47 531 M 15 8 1 1990 70 48 154 M 6 1 Odds Ratio (Log Scale) Favours Treatment Favours Control

Prophylactic Lidocaine in MI Outcome = death Favors treatment Favors placebo Relative risk (CI) Cumulative Year # RCTs Subjects 0.5 1 1.5 2 Recommendations Yes No Not mentioned 1970 2 304 9 1 1 1974 9 1451 8 0 2 1976 11 1686 5 0 2 This slide shows a cumulative meta-analysis of the effect of prophylactic lidocaine in preventing death from myocardial infarction. As in the previous examples, this slide shows: Expert opinion differs from available evidence Expert opinion varies 1978 12 1986 8 0 3 1985 14 8412 14 4 6 1988 15 8745 4 2 1 [Gordon - it was not clear in your original slide when the 1st meta-analysis was published. I have identified Hine et al, Arch Intern Med 1989, is this correct?] 1989 - 1st meta-analysis published

What went wrong?

Unclear too broad question Unrepresentative articles Failure to understand evidence quality Biased inferences

unclear too broad question unrepresentative articles Evidence quality poor understanding biased inferences explicit eligibility (PICO, methods) comprehensive search RoB assessment duplicate eligibility, risk of bias MA (absolute) Formal quality rating

unclear too broad question unrepresentative articles Evidence quality poor understanding biased inferences explicit eligibility (PICO, methods) comprehensive search RoB assessment duplicate eligibility, risk of bias MA (absolute) Formal quality rating

unclear too broad question unrepresentative articles Evidence quality poor understanding biased inferences explicit eligibility (PICO, methods) comprehensive search RoB assessment duplicate eligibility, risk of bias MA (absolute) Formal quality rating

unclear too broad question unrepresentative articles Evidence quality poor understanding biased inferences explicit eligibility (PICO, methods) comprehensive search RoB assessment duplicate eligibility, risk of bias MA (absolute) Formal quality rating

The right question all cancer therapy for all cancers all antiplatelet agents for all atheroembolic events (heart, head, leg) all aspirin doses for stroke 30 to 300 mg. for ischemic stroke How did you decide when ok to pool?

What were your criteria? Across range of patients interventions comparators outcomes Effect more or less same If not big effect in severe patients, no effect in mild big effect in high dose, no effect in low big effect in short term, none in long term

Inconsistency When doing meta-analysis, need to check if assumption is accurate: effect similar across patients interventions outcomes methodology

Are you happy pooling?

Are you happy pooling?

What criteria were you using? similarity of point estimates less similar, less happy overlap of confidence intervals less overlap, less happy

Homogenous Ho: RR1 = RR2 = RR3 = RR4 test for heterogeneity what is the p-value? p=0.99 for heterogeneity

-40 -24 -8 8 24 40 56 RRR (95% CI)

Heterogeneous test for heterogeneity what is the p-value? p-value for heterogeneity < 0.001

Only a little concerned I2 Interpretation 100% Why are we pooling? Very concerned Only a little concerned Getting concerned 0% No worries

Homogenous What is the I2 ? p=0.99 for heterogeneity I2=0%

Heterogeneous What is the I2 ? I2=89% p-value for heterogeneity < 0.001 I2=89%

Homogenous If this result, what next? p=0.99 for heterogeneity I2=0%

Heterogeneous If this result, what next? I2=89% p-value for heterogeneity < 0.001 I2=89%

Heterogeneity look for explanation patients interventions outcomes risk of bias No good explanation? What to do? Decrease confidence in effect estimates

Stroke p=0.99 for heterogeneity I2= 0%

Total Fractures

p=0.04 for heterogeneity

p=0.04 for heterogeneity I2=43%

Vitamin D versus placebo/control

Vitamin D versus placebo/control p= 0.07 for heterogeneity

Vitamin D versus placebo/control p= 0.07 for heterogeneity I2= 53%

p= 0.32 for heterogeneity

p= 0.32 for heterogeneity I2= 14%

Summary Lots of reasons traditional reviews went wrong Systematic reviews: strategies to protect against misleading results Single estimate most useful when same effect across patients, interventions, outcomes, methods Is there excessive heterogeneity? estimates too variable, confidence intervals non-overlapping low heterogeneity p-value, high I2 if so, look for explanation patients, intervention, outcome, methodology unexplained rate down for inconsistency Do you tweet?

@EBCPMcMaster Follow us on Twitter