1. Visual funnel plot inference for meta-analysis
Michael Kossmeier, Ulrich S. Tran & Martin Voracek
Department of Basic Psychological Research and Research Methods, School of Psychology, University of Vienna, Austria

2. The funnel plot
Significance contours: Studies falling in the light blue area have p values > 0.05
Significance contours: Studies falling in the dark blue area have p values < 0.05
Significance contours: Studies falling in the white area have p values < 0.01
Lower limit of 95% region
Upper limit of 95% region
Summary effect

3. The funnel plot
Studies scatter symmetrically around the summary effect
Studies with small or opposite effects seem to be missing
Studies with larger standard errors seem to report larger effect sizes on average

4. The funnel plot
Studies scatter symmetrically around the summary effect
Studies with small or opposite effects seem to be missing
Studies with larger standard errors seem to report larger effect sizes on average
However, funnel plot based conclusions subjective and often wrong (e.g., Terrin, Schmid & Lau (2005) Journal of clinical epidemiology; Simmonds (2015) Systematic Reviews)
Interpreted as indicative for publication bias when there is none (Type I error)
Failure to identify (simulated) publication bias (Type II error)

5. Visual inference
First proposed by Buja et al. (2009) as inferential framework to test if graphically displayed data do or do not support a hypothesis
Procedure:
1) Display your data and plots of data simulated under the null hypothesis simultaneously in so called lineups
2) Reject the null hypothesis if your data is identifiable out of all null plots in the lineup (α-level: 1 #plots in the lineup )
Visual inference empirically showed promising results when compared to conventional statistical tests (e.g., Majumder, Hofmann, & Cook (2013) American Statistician; Loy, Follett, & Hofmann (2016) American Statistician)
Helpful general purpose functions for visual inference are available within the R package nullabor (Wickham, Chowdhury & Cook 2014)

6. Visual inference – Lineup Example Q-Q plots (see, Adam Loy, Lendie Follett & Heike Hofmann (2016) Variations of Q–Q Plots: The Power of Our Eyes! The American Statistician)

7. Visual funnel plot inference
Funnel plots are a prime candidate field for the application of visual inference to increase the (often low) validity of funnel plot based conclusions
Controls α level by design
Generally applicable and many uses (assessment of publication bias, heterogeneity, outliers, subgroup effects)
Conventional statistical publication bias tests have limitations (low power and an increased α-error in a number of situations) (see, e.g., Sterne, Gavaghan, & Egger (2000) Journal of Clinical Epidemiology; Rücker, Schwarzer, & Carpenter (2008) Statistics in Medicine)
 Only if observed meta-analytic data is visually distinguishable from null data, conclusions based on the funnel plot might be warranted

8. Visual funnel plot inference with function funnelinf from R package metaviz
funnelinf(x, group = NULL, group_permut = TRUE, n = 20, y_axis = "se", null_model = "FEM", contours = TRUE, sig_contours = TRUE, trim_and_fill = FALSE, trim_and_fill_side = "left", egger = FALSE, show_solution = FALSE)
funnelinf provides features tailored for visual inference with funnel plots
Input: A data.frame with study effect sizes and standard errors
Allows null plot simulation under both classic meta-analysis models: The fixed effect model and the random effects model (Default: 19 null plots)

9. Function funnelinf from package metaviz: Features
funnelinf(x, group = NULL, group_permut = TRUE, n = 20, y_axis = "se", null_model = "FEM", contours = TRUE, sig_contours = TRUE, trim_and_fill = FALSE, trim_and_fill_side = "left", egger = FALSE, show_solution = FALSE)

10. Function funnelinf from package metaviz: Features
funnelinf(x, group = NULL, group_permut = TRUE, n = 20, y_axis = "se", null_model = "FEM", contours = TRUE, sig_contours = TRUE, trim_and_fill = TRUE, trim_and_fill_side = "left", egger = TRUE, show_solution = FALSE)

11. Shiny app: metaviz.shinyapps.io/funnelinf_app/

12. Visual funnel plot inference: Example meta-analaysis on the mozart effect (Pietschnig, J., Voracek, M., & Formann, A. K. (2010). Mozart effect-Shmozart effect: A meta-analysis. Intelligence, 38, )

13. Visual funnel plot inference: Example meta-analaysis on the mozart effect (Pietschnig, J., Voracek, M., & Formann, A. K. (2010). Mozart effect-Shmozart effect: A meta-analysis. Intelligence, 38, )

14. Visual funnel plot inference: Example meta-analaysis on the mozart effect (Pietschnig, J., Voracek, M., & Formann, A. K. (2010). Mozart effect-Shmozart effect: A meta-analysis. Intelligence, 38, )

15. Experimental pilot study – power to detect publication bias
Simulation of 90 meta-analyses under (strong) publication bias
5 expert raters assessed the 90 corresponding lineups (each with 20 plots)
6 experimental conditions (15 lineups each): Heterogeneity ( 𝐼 2 = 0% vs. 𝐼 2 = 50%) x Size (15, 30, 45 studies)
Comparison of visual funnel plot inference with two formal statistical tests based on funnel plot asymmetry
Egger’s regression test (Egger et al. (1997) Bmj)
Begg and Mazumdar rank correlation test (Begg & Mazumdar (1994) Biometrics)
Considered were only scenarios for which conventional funnel plot based tests are seen as appropriate (Ioannidis & Trikalinos (2007) Canadian Medical Association Journal)

16. Observed power – comparison of methods
Error bars are 𝐩 ± SE( 𝐩 )

17. Observed power – interrater differences

18. Conclusions and outlook
Visual funnel plot inference controls α-level by design and is generally applicable
Tailored software for visual funnel plot inference available (metaviz; shiny app)
First experimental results show lower power to detect publication than conventional tests (in a tailored scenario for conventional tests)
Interrater differences observed - “super-visual individuals”? (see, Majumder, Hofmann, & Cook (2013)  Journal of the American Statistical Association)
Further experimental research needed
more (different) raters (but fewer lineups per rater)
more conditions and different uses (heterogeneity assessment, subgroup effects,…)
benefit of additional displayed graphical information? (e.g., Egger‘s regression line, Trim and Fill,…)

19. Contact: michael.kossmeier@univie.ac.at
Thank you! Questions?
Contact:

20. Appendix: Publication bias simulation
Study effect size distribution:
𝐷 𝑖 ~ 𝑁(0.15, 0.4)
Publicaton bias

21. Appendix: Two examples where Egger’s regression test fails
Egger’s regression test p = 1.000
Egger’s regression test p = 0.997