1. RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
Funded through the ESRC’s Researcher Development Initiative
Session 1.3 – Equations
Meta-analysis
Prof. Herb Marsh
Ms. Alison O’Mara
Dr. Lars-Erik Malmberg
Department of Education,
University of Oxford

2. Steps in a meta-analysis
RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
Session 1.3 – Equations
2008
Steps in a meta-analysis
Establish research question
Define relevant studies
Develop code materials
Locate and collate studies
Pilot coding; coding
Data entry and effect size calculation
Main analyses
Supplementary analyses

3. RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
Assumptions 3

4. Fixed effects
In this and following formulae, we will use the symbols d and δ to refer to
any measure for the observed and the true effect size, which is not necessarily
the standardized mean difference.
Formula for the observed effect size in a fixed effects model
Where
dj is the observed effect size in study j
δ is the ‘true’ population effect
and ej is the residual due to sampling variance in study j

5. Calculating the observed effect size
To calculate the overall mean observed effect size (dj in the fixed effects equation)
where wi = weight for the individual effect size, and di = the individual effect size.

6. RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
Weighting
The effect sizes are weighted by the inverse of the variance to give more weight to effects based on large sample sizes
The standard error of each effect size is given by the square root of the sampling variance
SE =  vi
The variances are calculated differently for each type of effect size. 6 6 6

7. RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
Weighting
Variance for standardised mean difference effect size is calculated as
Where n1 = sample size of group 1, n2 is the sample size of group 2, and di = the effect size for study i.
Variance for correlation effect size is calculated as
Where ni is the total sample size of the study 7 7 7

8. Predictors in fixed effects model
Expand the general model to include predictors
Where
βs is the regression coefficient (regression slope) for the explanatory variable.
Xsj is the study characteristic (s) of study j.

9. Example of fixed effects model with predictor
Example: Gender as a predictor of achievement

10. Random effects
Formula for the observed effect size in a random effects model
Where
dj is the observed effect size in study j
δ is the mean ‘true’ population effect size
uj is the deviation of the true study effect size from the mean true effect size
and ej is the residual due to sampling variance in study j

11. Calculating the observed effect size
To calculate the overall mean observed effect size (dj in the random effects equation)
where wi = weight for the individual effect size, and di = the individual effect size.

12. Weighting in random effects models
RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
Weighting in random effects models
Random effects differs from fixed effects in the calculation of the weighting (wi)
The weight includes 2 variance components: within- study variance (vi) and between-study variance (vθ)
The new weighting for the random effects model (wiRE) is given by the formula:
vi is calculated the same as in the fixed effects models.
Recall the weighting
formula for fixed
effects model: 12

13. Weighting in random effects models
RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
Weighting in random effects models
vθ is calculated using the following formula
Where Q = Q-statistic (measure of whether effect sizes all come from the same population)
k = number of studies included in sample
wi = effect size weight, calculated based on fixed effects models. 13

14. Weighting in random effects models
RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
Weighting in random effects models
Thus, larger studies receive proportionally less weight in RE model than in FE model.
This is because a constant is added to the denominator, so the relative effect of sample size will be smaller in RE model 14

15. RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
Random effects models
RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
If the homogeneity test is rejected (it almost always will be), it suggests that there are larger differences than can be explained by chance variation (at the individual participant level). There is more than one “population” in the set of different studies.
The random effects model determines how much of this between-study variation can be explained by study characteristics that we have coded.

16. Predictors in random effects model
Expand the general model to include predictors
Where
βs is the regression coefficient (regression slope) for the explanatory variable.
Xsj is the study characteristic (s) of study j.

17. Example of random effects model with predictor
Example: Gender as a predictor of achievement

18. Multilevel modelling
Formula for the observed effect size in a multilevel model
Where
dj is the observed effect size in study j
0 is the mean ‘true’ population effect size
uj is the deviation of the true study effect size from the mean true effect size
and ej is the residual due to sampling variance in study j
Note: This model treats the moderator effects as fixed and the ujs as random effects.

19. Predictors in a multilevel model
In this equation, predictors are included in the model.
s is the regression coefficient (regression slope) for the explanatory variable. (Equivalent to β in multiple regression.)
Xsj is the study characteristic (s) of study j.

20. Example of multilevel model with predictor
Example: Gender as a predictor of achievement

21. Simplifying the multilevel equation
If between-study variance = 0, the multilevel model simplifies to the fixed effects regression model
If no predictors are included the model simplifies to random effects model
If the level 2 variance = 0 , the model simplifies to the fixed effects model

22. In practice...
Many meta-analysts use an adaptive (or “conditional”) approach
IF between-study variance is found in the homogeneity test
THEN use random effects model
OTHERWISE use fixed effects model

23. In practice...
Fixed effects models are very common, even though the assumption of homogeneity is “implausible” (Noortgate & Onghena, 2003)
There is a considerable lag in the uptake of new methods by applied meta-analysts
Meta-analysts need to stay on top of these developments by
Attending courses
Wide reading across disciplines

24. What do the models involve?
RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
What do the models involve? 24

25. Conducting fixed effects meta-analysis
RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
Conducting fixed effects meta-analysis
Usually start with a Q-test to determine the overall mean effect size and the homogeneity of the effect sizes (MeanES.sps macro)
If there is significant homogeneity, then:
1) should probably conduct random effects analyses instead
2) model moderators of the effect sizes (determine the source/s of variance)
It should be noted that within each type (i.e., ANOVA and multiple regression), there may be variations in the equations used in the analyses. For fixed effects, we refer to the Hedges & Olkin (1985) methods (see also Lipsey & Wilson, 2001) 25

26. Q-test of the homogeneity of variance
RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
Q-test of the homogeneity of variance
The homogeneity (Q) test asks whether the different effect sizes are likely to have all come from the same population (an assumption of the fixed effects model). Are the differences among the effect sizes no bigger than might be expected by chance?
di = effect size for each study (i = 1 to k)
= mean effect size
= a weight for each study based on the sample size
However, this (chi-square) test is heavily dependent on sample size. It is almost always significant unless the numbers (studies and people in each study) are VERY small. This means that the fixed effect model will almost always be rejected in favour of a random effects model.

27. RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
Fixed effects: mean effect size
2008
Significant heterogeneity in the effect sizes therefore random effects more appropriate and/or moderators need to be modelled 27

28. ANOVA
The analogue to the ANOVA homogeneity analysis is appropriate for categorical variables
Looks for systematic differences between groups of responses within a variable
Easy to implement using MetaF.sps macro
MetaF ES = d /W = Weight /GROUP = TXTYPE /MODEL = FE.

29. Multiple regression
Multiple regression homogeneity analysis is more appropriate for continuous variables and/or when there are multiple variables to be analysed
Tests the ability of groups within each variable to predict the effect size
Can include categorical variables in multiple regression as dummy variables
Easy to implement using MetaReg.sps macro
MetaReg ES = d /W = Weight /IVS = IV1 IV2 /MODEL = FE.

30. Conducting random effects meta-analysis
Like the FE model, RE uses ANOVA and multiple regression to model potential moderators/predictors of the effect sizes, if the Q- test reveals significant heterogeneity
Easy to implement using MetaF.sps macro (ANOVA) or MetaReg.sps (multiple regression).
MetaF ES = d /W = Weight /GROUP = TXTYPE /MODEL = ML.
MetaReg ES = d /W = Weight /IVS = IV1 IV2 /MODEL = ML.

31. Random effects: mean effect size
RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
Random effects: mean effect size
2008
Significant heterogeneity in the effect sizes therefore need to model moderators 31

32. Conducting multilevel model analyses
RDI Meta-analysis workshop - Marsh, O'Mara, & Malmberg
2008
Conducting multilevel model analyses
Similar to multiple regression, but corrects the standard errors for the nesting of the data
Start with an intercept-only (no predictors) model, which incorporates both the outcome-level and the study-level components
This tells us the overall mean effect size
Is similar to a random effects model
Then expand the model to include predictor variables, to explain systematic variance between the study effect sizes 32 32

33. Multilevel set-up
(MLwiN screenshot)

34. Multilevel: mean effect size
Using the same simulated data set with n = 15

35. Comparability of random and multilevel models (no predictors)

36. Van den Noortgate & Onghena (2003)
The random effects is better than the fixed effects approach in almost all conceivable cases
“The results of the simulation study suggest that the maximum likelihood multilevel approach is in general superior to the fixed-effects approaches, unless only a small number of studies is available. For models without moderators, the results of the multilevel approach, however, are not substantially different from the results of the traditional random- effects approaches” (p. 765)

37. Conclusions Multilevel models:
build on the fixed and random effects models
account for between-study variance (like random effects)
Are similar to multiple regression, but correct the standard errors for the nesting of the data. Improved modelling of the nesting of levels within studies increases the accuracy of the estimation of standard errors on parameter estimates and the assessment of the significance of explanatory variables (Bateman and Jones, 2003).
Multilevel modelling is more precise when there is greater between-study heterogeneity
Also allows flexibility in modelling the data when one has multiple moderator variables (Raudenbush & Bryk, 2002)

38. Other (potential) benefits of MLM
Multilevel modelling has the promise of being able to include multivariate data – still being developed
Easy to implement in MLwiN (once you know how!)
See worked examples for HLM, MLwiN, SAS, & Stata at

39. References
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage Publications.
Van den Noortgate, W., & Onghena, P. (2003). Multilevel meta-analysis: A comparison with traditional meta-analytical procedures. Educational and Psychological Measurement, 63,
Wilson’s “meta-analysis stuff” website:
Raudenbush, S.W. and Bryk, A.S. (2002). Hierarchical Linear Models (2nd Ed.).Thousand Oaks: Sage Publications.