Data Sciences Unit, School of Psychology, Deakin University Systematic Reviews and Meta-Analysis in Behavioural Medicine: A Practical Introduction to Best Practices Meta-analysis slides Dr Emily Kothe Data Sciences Unit, School of Psychology, Deakin University

Identification Screening Included Records identified through database searching (n = 6355) Additional records identified through other sources (n = 0) Identification Records after duplicates removed (n = 5724) Screening Records screened (n = 5724) Records excluded (n = 5703) Full-text articles assessed for eligibility (n = 21) Full-text articles excluded with reasons (n=9) Not industy appropriate (n=5) Passive intervention (n=2) Did not measure injury rates/safety behaviour (n=1) Full-text articles excluded, with reasons (n = 9 ) Eligibility Studies included in qualitative synthesis (n = 15) Included Studies included in quantitative synthesis (n =?)

Identification Screening Included Records identified through database searching (n = 6355) Additional records identified through other sources (n = 0) Identification Records after duplicates removed (n = 5724) Screening Records screened (n = 5724) Records excluded (n = 5703) Full-text articles assessed for eligibility (n = 21) Full-text articles excluded with reasons (n=9) Not industy appropriate (n=5) Passive intervention (n=2) Did not measure injury rates/safety behaviour (n=1) Full-text articles excluded, with reasons (n = 9 ) Eligibility Studies included in qualitative synthesis (n = 15) Included Studies included in quantitative synthesis (n =?)

Meta-analysis Provides a quantitative summary of the results included studies Provides a mechanism for statistically examining variation in effect sizes across studies

Decide on the appropriate effect size When developing the protocol you should identify the effect size(s) that are most appropriate to the research question Within behavioural medicine commonly used effect sizes include Odds ratios Correlations Standardised mean differences

Data for odds ratios 2 x 2 frequency table Odds ratio and confidence intervals Risk difference Risk ratio And much more…

Data for correlations r and N r and SE r and variance Fisher’s Z and N Fisher’s Z and SE r and t value t value and sample size (for correlation) p value and sample size (for correlation)

Data for standardised mean differences Mean, SD and N for each group d and confidence interval d and variance p value and n And much more…

Compiling effect sizes - A perfect world In the perfect world where all studies are well described Two reviewers extract effect size data from each paper, extraction is compared and disagreements resolved through consensus Each reviewer records Data required to calculate the effect size Test statistics (e.g. p values, t values) relevant to the effect size Assumptions made when extracting data

Best practice recommendations “Facilitate cumulative science by future-proofing meta-analyses: Disclose all meta-analytic data (effect sizes, sample sizes for each condition, test statistics and degrees of freedom, means, standard deviations, and correlations between dependent observations) for each data point. Quote relevant text from studies that describe the meta- analytic data to prevent confusion, such as when one effect size is selected from a large number of tests reported in a study. When analyzing subgroups, include quotes from the original study that underlie this classification, and specify any subjective decisions.” Lakens, Hilgard and Staaks (2016)

Best practice recommendations “Facilitate quality control: Specify which effect size calculations are used and which assumptions are made for missing data (e.g., assuming equal sample sizes in each condition, imputed values for unreported effect sizes), if necessary for each effect size extracted from the literature. Specify who extracted and coded the data, knowing it is preferable that two researchers independently extract effect sizes from the literature” Lakens, Hilgard and Staaks (2016)

Minimum best practice data extraction for correlations Study ID r N Exact P value Location in Text Notes

Minimum best practice data extraction for standardised mean difference Study ID Means group 1 N group 1 SD group 1 Means group 2 N group 2 SD group 2 Exact p value Location in Text Notes

Minimum best practice data extraction for standardised mean difference Study ID N events group 1 N non-events group 1 N events group 2 N non-events group 2 Exact p value Location in Text Notes

Compiling effect sizes – An imperfect world In the perfect world all studies are well described Sadly, we don’t live in a perfect world Studies commonly do not report information in the format that you would find most helpful You may need to use a number of formulas different to calculate effect sizes based on the information provided Even after trying to extract maximal data from all studies you’ll often need to contact authors

Data availability: An example Data type N % Events and sample size in each group 150 31.65% Odds ratio and confidence limits 137 28.90% Event rate and sample size in each group 84 17.72% This data is not available 57 12.03% Risk ratio and confidence limits 21 4.43% Events and non-events in each group 13 2.74% Chi-squared and total sample size 10 2.11% Log odds ratio and variance 2 0.42% Risk different and standard error 0.00% Risk difference and variance Non-events and sample size in each group Risk difference and confidence limits Log risk ratio and variance Log odds ratio and standard error Log risk ratio and standard error Peto's O-E and V Total 474 100.00% Of 474 categorical effects identified in a recent analysis, only 2.74% reported data in the format we wanted 12% of studies did not report required data Need systematic methods for identifying and then converting effect sizes

Compiling effect sizes – Your options Depending on the scope of your analysis and the software you’re using you have a number of options CMA (Comprehensive Meta-Analysis) allows direct entry of 100 different data formats Online calculators can be used to hand calculate effect sizes using different data formats You can use R packages to compile effect sizes to prepare for analysis

Compiling effect sizes – Comparison of options Advantages Disadvantages CMA Easy to compile effect sizes in different formats Expensive Meta-analysis is relatively “black box” Online calculators Free Allows for very specialised effect size calculations that aren’t built into any software Very time consuming if you have a lot to calculate Can be difficult to reproduce your calculation R packages Reproducible Intimidating if you’re not familiar with R

Compiling effect sizes – metafor example See the “Compile dataset.rmd” for a worked example This requires the metafor package is installed

Compiling effect sizes – An imperfect world REMEMBER It is especially important to use best practice to report how you extracted effect sizes when studies are not perfectly reported. You will (often) need to make assumptions when extracting data and it must be clear to readers (and future you) what those were so your meta-analysis can be replicated

Choosing your model Most meta-analytic software packages incorporate two statistical models for meta-analysis, the fixed-effect model and the random- effects model. You should choose which one you will use in the protocol stage

Fixed effect vs. Random effects meta-analysis Fixed effect meta-analysis assumes that the underlying true effect does not vary between studies and that variations in observed effects are due to sampling error Random effects meta-analysis assumes that the true effect does vary between studies within the meta-analysis and that variation in observed effects are due to sampling error and by moderators of the effect

Fixed effect vs. Random effects meta-analysis Increasingly, meta-analysts recommend that random effects meta- analysis be the default analysis Because The assumptions underlying the fixed effect model are almost never supported Using a random effects model when the fixed effect model is true gives the same results as running the fixed effect model but running a FE model when random effects is true gives misleading results

Running your meta-analysis Meta-analysis requires attention to detail and is very time consuming. But once data is compiled in the format you need not very difficult.

Running your meta-analysis Work through “SMD meta analysis template.rmd” and “Correlation meta analysis template.rmd” for examples of how to conduct meta- analysis of two common types of data.

Useful R packages metagear metafor Compute.es The abstract screener function and plot_PRISMA are helpful. Note that this package has a lot of dependencies for functions you might not care about, more advanced R users might want to just download the scripts for the functions you need (they’re on github) metafor Good all-round package for running meta-analysis and meta-regression. Compute.es As the name implies, this package has a range of functions for computation of effect sizes

“Systematic Reviews and Meta-Analysis in Behavioural Medicine: A Practical Introduction to Best Practices” by Emily Kothe is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. These slides were created in December 2016 for presentation at the International Congress of Behavioral Medicine. These slides and up to date versions of associated files are available at: 10.17605/OSF.IO/6BK7B