1. Conducting Meta-Analysis in R: A Practical Introduction
Emily Kothe & Mathew Ling
School of Psychology
Data Sciences Unit
Deakin University
While you wait, please go to RStudio and enter: install.packages(beepr); beepr::beep()

2. Presenters School of Psychology Data Sciences Unit
Mathew Ling
Emily Kothe
@lingtax
@emilyandthelime
School of Psychology Data Sciences Unit

3. Background Meta-analysis:
Provides a quantitative summary of the results included studies
Provides a mechanism for statistically examining variation in effect sizes across studies
This workshop is intended to provide a practical introduction to conducting meta- analyses in R, principally using pre-templated markdown scripts.

4. Meta-analysis overview
Meta analysis is about synthesis of primary findings
Garbage in, garbage out
Weights studies by the variance of effect size estimate
Less variance = More weight
Necessarily concerned with Effect sizes not p-values
E.g. Mean differences, Odds ratios, correlations
library(metafor) example(forest.rma)

5. Meta-analysis overview
Two general types
Fixed effects
Random effects
Fixed-effects meta-analysis
Looking for the common effect
Differences across studies caused by random variation
Random-effects meta-analysis
Looking for average effect
True differences in effects plus random variation and
library(metafor) example(forest.rma)

6. Background
This workshop focuses on use of metafor (Viechtbauer, 2010) however Data Science members can also advise on
A number of other R meta-analysis packages:
Psychometric, robumeta
Meta-analysis using other software:
stata and CMA

7. Overview R Setup and package installation Run a meta-analysis in R
RMarkdown Templates
Preparing for a meta-analysis
Data extraction
Effect size calculation
Generalising to new projects

8. R Setup
Creating a Project
Identifying setup issues.

9. Live coding a meta-analysis Learning the RStudio interface via doing
Live code a meta-analysis first:
library(metafor)
dat <- read.csv("data_withES.csv")
head(dat)
output <- rma(yi, vi, data=dat)
output
forest(output)
Conduct a meta-analysis using and the data.csv file
Conduct a meta-analysis using and gibson2002.csv file (note this requires changing names of columns in lines 6-12)

10. Now it’s your turn Learning the RStudio interface via doing
Live code a meta-analysis first:
library(metafor)
dat <- read.csv("data_withES.csv")
head(dat)
output <- rma(yi, vi, data=dat)
output
forest(output)
Conduct a meta-analysis using and the data.csv file
Conduct a meta-analysis using and gibson2002.csv file (note this requires changing names of columns in lines 6-12)

11. Introduction to rMarkdown
A Tale of Two Scripts

12. Reporting errors are common
Nuijten et al. (2015)
Audit of 250,000 published NHST between
“...half of all published psychology papers that use NHST contained at least one p-value that was inconsistent with its test statistic and degrees of freedom…”
“...One in eight papers contained a grossly inconsistent p-value that may have affected the statistical conclusion.”

13. .R vs .Rmd .R files - Person to computer communication
Nothing but the code
Useful for communicating tools / plans
Limited at communicating knowledge
.Rmd files - Person to Person communication
Text AND Code (in ‘Chunks’)
Eliminates analysis-reporting gap
Allows reproducible reports / templates
Focus just on content
Inefficient for passing code-heavy helpers
Just write packages!

14. Why markdown templates for meta-analysis particularly
Most meta-analyses have common structures
Inputs: effect size and SE per study
Outputs: MA estimate, forest plot, funnel plot
So scripting each time is reinventing the wheel
Templates also:
Provide guidance for interpretation and reporting.
Reduce barriers to entry for inexperienced R users.

15. Let’s run a meta-analysis using rMarkdown
Open RStudio
Open the file “Basic_SMD_Meta-Analysis.Rmd”
Click “Knit”

16. Popping the hood
This analysis script is completing several steps in order to generate this report
Initiating all the R packages we need for our analysis
Reading in the most current version of a spreadsheet containing extracted data
Calculating effect sizes
Running a random effects meta-analysis
Reporting the model statistics (effect size and heterogeneity tests)
Generating a forest plot
Generating a funnel plot

17. Making data.csv
This will introduce the data extraction and data cleaning process we’re using

18. Let’s practice extracting some data
We are investigate the impact of exercise on anxiety among individuals with cancer relative to a no-intervention or waitlist control. Our effect size will be standardised mean difference (SMD).
Your task (if you choose to accept it) is to extract post-intervention anxiety outcomes from these studies.

19. Data extraction
Generally you’d do this into a spreadsheet (templates here:
Because we all want to extract to the same sheet, we’ve put ours on googledocs and provided a form to give you access.
You should:
Go to our data extraction form [
Enter study ID as Surname of first author, underscore, and year. e.g. Smith_2015
Extract mean, SD, N for each group (intervention & control)

20. Checking our extraction

21. Best practice data extraction
This will introduce the data extraction and data cleaning process we’re using

22. Best practice extraction in a perfect world...
Two reviewers extract effect size data from each paper, extraction is compared and disagreements resolved through consensus Each reviewer records
Data required to calculate the effect size
Test statistics (e.g. p values, t values) relevant to the effect size
Assumptions made when extracting data

23. Minimum best practice data extraction for standardised mean difference
Study ID
Means group 1
N group 1
SD group 1
Means group 2
N group 2
SD group 2
Exact p value
Location in Text
Notes

24. Minimum best practice data extraction for correlations
Study ID r N
Exact p value
Location in Text
Notes

25. Minimum best practice data extraction for odds ratio
Study ID
N events group 1
N non-events group 1
N events group 2
N non-events group 2
Exact p value
Location in Text
Notes

26. Best practice extraction in an (im)perfect world...
In the perfect world all studies are well described Sadly, we don’t live in a perfect world Studies commonly do not report information in the format that you would find most helpful
You may need to use a number of formulas different to calculate effect sizes based on the information provided
Even after trying to extract maximal data from all studies you’ll often need to contact authors

27. Data availability: An example
Of 474 categorical effects identified in a recent analysis, only 2.74% reported data in the format we wanted 12% of studies did not report required data We need systematic methods for identifying and then converting effect sizes

28. REMEMBER
It is especially important to use best practice to report how you extracted effect sizes when studies are not perfectly reported. You will (often) need to make assumptions when extracting data and it must be clear to readers (and future you) what those were so that your meta-analysis can be replicated

29. Calculating effect sizes
This will introduce the data extraction and data cleaning process we’re using

30. Let’s practice compiling data from an incomplete dataset
We are going to run “CompileR.Rmd” to look at how we can compile effect sizes when we have different statistics available
Open the file “CompileR.Rmd”
Click “Knit”

31. Conducting (random effects) meta-analysis
This will introduce the data extraction and data cleaning process we’re using

32. Let’s practice a meta-analysis
We’re going to step through re-calibrating the template together.
Pay attention, because you’re going to do it yourselves next.

33. Verbose Meta-Analysis
This is where markdown gets really cool

34. Sidenotes...
If you want to extract the R code from any of the Rmd files we’ve discussed today you can use the following code to do so
library(knitr)
purl("CompileR.Rmd")

35. Sidenotes...
All code and slides underlying this workshop are available at GitHub ( and OSF ( Those resources get updated when we add new features.