1. Department of O UTCOMES R ESEARCH

2. Daniel I. Sessler, M.D. Michael Cudahy Professor and Chair Department of O UTCOMES R ESEARCH The Cleveland Clinic Clinical Research Design Systematic Reviews and Meta-analysis

3. Literature Reviews Reviews are important Often too much primary literature Clinicians cannot critically review all literature Classical reviews Informed synthesis by authors –Most helpful when authors are experts and active investigators Excellent perspective –Integrates historical development with future directions Typically restricted to best relevant articles Most suitable for reviewing an entire field Subject to author(s) bias

4. Useful for specific interventions & outcomes Specific, important, and sensible question essential Equally effective for complications and therapeutic outcomes Standardized search of all relevant work Documented and reproducible selection process Tabular presentation, often stratified by –Research approach –Study quality –Population –Outcome Synthesis can be Qualitative, based on authors’ expertise (and bias) Quantitative: meta-analysis Systematic Reviews

5. Meta-analysis Statistical summary of systematic review Effect size and significance Patient level (patient pooling) or study level (aggregate stats) –Individual patient data preferable, but rarely available Usually used for randomized trials Can be used for observational studies— with great caution Studies must evaluate similar treatment & outcomes Suitable for various types of data Dichotomous, continuous, risk difference, relative risk, etc. Generalizability good; internal validity variable

6. Data-acquisition Individual studies are unit of analysis Summary statistics are the data elements Consider studies to be like patients in a trial Rigorous a priori inclusion and exclusion criteria Specify search strategies and sources of studies Which databases? Search terms? Language restrictions? Randomized trials only? Primary outcomes only? Published versus unpublished? Specify adjudication methods

7. Sample Data-extraction Form Population Comparison Treatment Active vs. placebo Outcome(s) Measures of quality Surprisingly difficult Adjudication critical

8. Evaluating Study Quality No good way Many design errors non-obvious or subtle Various scoring systems used; points for Legitimate randomization Concealed allocation Blinded outcome evaluation Drop-outs and reasons described Standard-of-care: report quality of included studies

9. Reporting Search Results

10. Major Sources of Error Garbage in, garbage out Meta-analysis never better than underlying studies Cannot correct for methodologic errors or bias Reporting bias Changed or omitted primary outcomes Significant findings 2.2-4.7 X more likely to be complete (Dwan 2008) Subtle (or not) treatment & measurement differences Publication bias Large trials are almost always published Positive studies usually published even if under-powered Small negative studies less likely than others to be published –Censoring by authors or corporate sponsors –Appropriate editorial decision, but unpublished studies disappear –Meta-analysis depends on knowing about all relevant results

11. Funnel Plots SE of Log(OR) Log(OR)

12. Heterogeneity Data: variation in study results exceeding chance Biology: true differences related to methodology Differences in populations: age, gender, ethnicity, etc. Differences in drug dose (or drug within a class) Unappreciated patient factors Tests: chi square, etc. Analysis strategies Minor heterogeneity –Report amount –Combined analysis may be sensible Treat serious heterogeneity as an interaction –Stratify analysis as for other effect modifiers

13. Analysis Strategies Fixed-effects model Assumes all trials share same underlying treatment effect –Treats each trial as random samples from one large trial –Differences in results due to chance alone Similar to Mantel-Haenszel Often over-estimates significance Random-effects model Assumes each study estimates a unique treatment effect –That is, may truly differ from other included studies –Allows heterogeneity, and is required for heterogeneous data Weights smaller studies more heavily Generally provides similar effect estimate with lower precision –More conservative; probably should always be used

14. Forest Plots Log weighted mean effect ≈ sum of {log (effect)/variance)} for individual studies, divided by sum of 1/variance

15. How Good are Meta-analyses? “Large” defined by n≥1,000“Large” defined by power Generally, pretty good. But not perfect. Cappelleri, JAMA 1996

16. Meta-analyses Increasingly Common Most published as part of systematic reviews Increasingly included in trial reports Objective comparison of current to previous results Grant applications Summarize knowledge Support equipoise Need for proposed trial Complications unlikely Blood loss with low- dose perioperative aspirin

17. Cochrane Collaboration International non-profit, 1993 Repository for meta-analyses Standardized reporting QUORUM (1999) PRISMA (2009) Provides free software Evidence-based med movement David Sackett Gordon Guyatt Tom Chalmers Archie Cochrane

18. Summary Systematic reviews More objective than “expert” reviews May lack expert perspective and subtlety Meta-analysis is quantification of systemic review Subject to major errors Any problems with underlying studies remain Publication and reporting bias can be substantial Heterogeneity can complicate analysis Conduct and report per guidelines Useful summary of available literature Especially when many similar studies are available

19. Department of O UTCOMES R ESEARCH