1. Simon Thornley Meta-analysis: pooling study results

2. Objective Understand the philosophy of meta-analysis and its contribution to epidemiology and science. Understand the limitations of meta-analysis

3. Introduction Systematic quantitative integration of results several independent studies Distinct from a narrative review “expert” Synthesis of published information. Usually considered only appropriate for RCTs Still controversial even in this context. Google search on “meta-analysis” 8 million hits!

4. Criticism “statistical alchemy” for the 21st Century “ The intellectual allure of making mathematical models and aggregating collections of studies has been used as an escape from the more fundamental scientific challenges” -Feinstein.

5. Purposes of meta analysis Inefficiency of traditional narrative reviews. Allow researchers to keep abreast of accumulating evidence Resolution of uncertainty when research disagrees? Increase statistical power, enhances precision of effect estimates – especially small effects Allows exploratory analysis (subgroups)

6. Inadequate sample size? (Deal with type-2 error) Single trials too small to detect moderate effects (low power – high chance of Type-2 error (false- negative)) Investigators often over enthusiastic about size of treatment effects and sample size Meta-analysis doesn’t deal with other threats to study validity (bias, measurement error; in fact, may increase) e.g. CVD death vs. total mortality

7. Accept H 0 Reject H 0 Statistical Test result H0H0 True False OK Type-1 error Type-2 error Prob of a type 1 error = alpha  usually fixed, say 0.05  Prob of a type 2 error = beta  = 1-power

8. Random error lecture Average odds ratio is 21?? Consistency??

9. Which studies? Need defined question, state MESH terms Reproducible Exhaustive search Unpublished and published studies Variety of databases.

10. Difference in means, Standardized differences in means Survival measures Relative risk Odds ratio Risk difference NNT [=1/RD] Incidence rate ratios (person time data) Typical summary outcome measures Binary:Continuous:

11. Assume distribution of true effects Aim is to measure mean of distribution of true effects Greater heterogeneity --> greater variation Gives greater weight to small studies than fixed effect method of analysis. More conservative (wider confidence interval around effect estimate, compared to fixed effect method) Mantel-Haenszel method treat each trial as a “stratum” take weighted average of effects. O - E (Peto) method Binary outcome (e.g. death) O i =observed # deaths on treatment in trial i E i =expected # deaths (assuming no treat effect) look at average of O i - E i over all trials Assumes underlying true effect for each study and differences only due to random error Methods of analysis Fixed effectRandom effect

12. Dietary fat and cholesterol

13. Reduced or modified dietary fat and all- cause mortality

14. Publication bias

15. When meta-analysis goes bad… In CVD drug research, CVD outcomes often favoured over total mortality Which would you prefer????

16. Publication bias: other methods Ioannidis JPA, Trikalinos TA. An exploratory test for an excess of significant findings. Clin. Trials 2007;4(3):245-53. Calculate expected number of positive studies, given: Sample size of individual studies Number of events in controls Summary effect (assumed true)

17. Statin meta-analysis

18. Problems Combining heterogeneous studies (apples and oranges) Combining good and bad studies (good and bad apples) (study quality) Publication bias (tasty apples only) The "Flat Earth" criticism – reductionism –(Braeburns only) Combining data (individual v summary data stewed apples have different character to raw) Application to randomized studies only? Type-2 error only one problem with epi studies

19. Meta analysis in observational studies MA often applied in observational studies As often as RCTs (Egger et al) …. with controversy …. Confounding and bias unlikely to “cancel out” Publication bias and “research initiation bias” (i.e. studies only done when there is an association) Different ways of reporting/analysing result (e.g different outcome measures, confounders, models, exposure levels)

20. Summary Meta-analyses increasingly used Logical only for RCTs? Summarise medical literature Reduce type-2 error by increasing sample size. Don’t deal with other types of epidemiological error (confounding/measurement error) Prone to unique type of error (Publication bias) Can be difficult to detect