Visual funnel plot inference for meta-analysis Michael Kossmeier, Ulrich S. Tran & Martin Voracek Department of Basic Psychological Research and Research Methods, School of Psychology, University of Vienna, Austria

The funnel plot Significance contours: Studies falling in the light blue area have p values > 0.05 Significance contours: Studies falling in the dark blue area have p values < 0.05 Significance contours: Studies falling in the white area have p values < 0.01 Lower limit of 95% region Upper limit of 95% region Summary effect

The funnel plot Studies scatter symmetrically around the summary effect Studies with small or opposite effects seem to be missing Studies with larger standard errors seem to report larger effect sizes on average

The funnel plot Studies scatter symmetrically around the summary effect Studies with small or opposite effects seem to be missing Studies with larger standard errors seem to report larger effect sizes on average However, funnel plot based conclusions subjective and often wrong (e.g., Terrin, Schmid & Lau (2005) Journal of clinical epidemiology; Simmonds (2015) Systematic Reviews) Interpreted as indicative for publication bias when there is none (Type I error) Failure to identify (simulated) publication bias (Type II error)

Visual inference First proposed by Buja et al. (2009) as inferential framework to test if graphically displayed data do or do not support a hypothesis Procedure: 1) Display your data and plots of data simulated under the null hypothesis simultaneously in so called lineups 2) Reject the null hypothesis if your data is identifiable out of all null plots in the lineup (α-level: 1 #plots in the lineup ) Visual inference empirically showed promising results when compared to conventional statistical tests (e.g., Majumder, Hofmann, & Cook (2013) American Statistician; Loy, Follett, & Hofmann (2016) American Statistician) Helpful general purpose functions for visual inference are available within the R package nullabor (Wickham, Chowdhury & Cook 2014)

Visual inference – Lineup Example Q-Q plots (see, Adam Loy, Lendie Follett & Heike Hofmann (2016) Variations of Q–Q Plots: The Power of Our Eyes! The American Statistician)

Visual funnel plot inference Funnel plots are a prime candidate field for the application of visual inference to increase the (often low) validity of funnel plot based conclusions Controls α level by design Generally applicable and many uses (assessment of publication bias, heterogeneity, outliers, subgroup effects) Conventional statistical publication bias tests have limitations (low power and an increased α-error in a number of situations) (see, e.g., Sterne, Gavaghan, & Egger (2000) Journal of Clinical Epidemiology; Rücker, Schwarzer, & Carpenter (2008) Statistics in Medicine)  Only if observed meta-analytic data is visually distinguishable from null data, conclusions based on the funnel plot might be warranted

Visual funnel plot inference with function funnelinf from R package metaviz funnelinf(x, group = NULL, group_permut = TRUE, n = 20, y_axis = "se", null_model = "FEM", contours = TRUE, sig_contours = TRUE, trim_and_fill = FALSE, trim_and_fill_side = "left", egger = FALSE, show_solution = FALSE) funnelinf provides features tailored for visual inference with funnel plots Input: A data.frame with study effect sizes and standard errors Allows null plot simulation under both classic meta-analysis models: The fixed effect model and the random effects model (Default: 19 null plots)

Function funnelinf from package metaviz: Features funnelinf(x, group = NULL, group_permut = TRUE, n = 20, y_axis = "se", null_model = "FEM", contours = TRUE, sig_contours = TRUE, trim_and_fill = FALSE, trim_and_fill_side = "left", egger = FALSE, show_solution = FALSE)

Function funnelinf from package metaviz: Features funnelinf(x, group = NULL, group_permut = TRUE, n = 20, y_axis = "se", null_model = "FEM", contours = TRUE, sig_contours = TRUE, trim_and_fill = TRUE, trim_and_fill_side = "left", egger = TRUE, show_solution = FALSE)

Shiny app: metaviz.shinyapps.io/funnelinf_app/

Visual funnel plot inference: Example meta-analaysis on the mozart effect (Pietschnig, J., Voracek, M., & Formann, A. K. (2010). Mozart effect-Shmozart effect: A meta-analysis. Intelligence, 38, 314-323.)

Visual funnel plot inference: Example meta-analaysis on the mozart effect (Pietschnig, J., Voracek, M., & Formann, A. K. (2010). Mozart effect-Shmozart effect: A meta-analysis. Intelligence, 38, 314-323.)

Visual funnel plot inference: Example meta-analaysis on the mozart effect (Pietschnig, J., Voracek, M., & Formann, A. K. (2010). Mozart effect-Shmozart effect: A meta-analysis. Intelligence, 38, 314-323.)

Experimental pilot study – power to detect publication bias Simulation of 90 meta-analyses under (strong) publication bias 5 expert raters assessed the 90 corresponding lineups (each with 20 plots) 6 experimental conditions (15 lineups each): Heterogeneity ( 𝐼 2 = 0% vs. 𝐼 2 = 50%) x Size (15, 30, 45 studies) Comparison of visual funnel plot inference with two formal statistical tests based on funnel plot asymmetry Egger’s regression test (Egger et al. (1997) Bmj) Begg and Mazumdar rank correlation test (Begg & Mazumdar (1994) Biometrics) Considered were only scenarios for which conventional funnel plot based tests are seen as appropriate (Ioannidis & Trikalinos (2007) Canadian Medical Association Journal)

Observed power – comparison of methods Error bars are 𝐩 ± SE( 𝐩 )

Observed power – interrater differences

Conclusions and outlook Visual funnel plot inference controls α-level by design and is generally applicable Tailored software for visual funnel plot inference available (metaviz; shiny app) First experimental results show lower power to detect publication than conventional tests (in a tailored scenario for conventional tests) Interrater differences observed - “super-visual individuals”? (see, Majumder, Hofmann, & Cook (2013)  Journal of the American Statistical Association) Further experimental research needed more (different) raters (but fewer lineups per rater) more conditions and different uses (heterogeneity assessment, subgroup effects,…) benefit of additional displayed graphical information? (e.g., Egger‘s regression line, Trim and Fill,…)

Contact: michael.kossmeier@univie.ac.at Thank you! Questions? Contact: michael.kossmeier@univie.ac.at

Appendix: Publication bias simulation Study effect size distribution: 𝐷 𝑖 ~ 𝑁(0.15, 0.4) Publicaton bias

Appendix: Two examples where Egger’s regression test fails Egger’s regression test p = 1.000 Egger’s regression test p = 0.997