1. (1.7) Modeling Heterogeneity; Interpreting evidence (Part II)
Professor James S. Kim
S32-Accumulating Evidence

2. Agenda Part I Part II – Activity
Review: modeling heterogeneity in meta-analysis
Conduct moderator analysis of Kim & Quinn (2013) using random effects models
Replication in STATA and Excel
Part II – Activity
Readings: Durlak et al. paper and codebook and APA-MARS standards – outline and model for my paper (I’ll show you how we used our model to organize our outline)
Exercise 3 – directions and goals
Independent Study  Presentation  Final Paper (1-1 meeting)
Module 2 – syllabus updates

3. Goals
Understand how to generate separate estimates of τ2 (between study variance component) when pooling means effects by subgroup (NB – use fixed effect models to generate the estimates)
Replicate STATA results in Excel
Be able to use your “model” meta-analysis to outline your own study (I will use Durlak et al. 2011)

4. Modeling heterogeneity in meta-analysis
Three core questions plus 1 more question
(1) Is there a common, fixed effect? Is there evidence of heterogeneity in true effects(Q)
(2) If we reject the null hypothesis of homogeneous effects, how much do the mean effects vary across studies—i.e., what is the between study variance? (τ2)
(3) If we reject the null hypothesis of homogeneous effects, what’s the true study-to-study heterogeneity in effects? (I2)
(4) If we use a random effects model to pool mean effects by subgroup, how do we estimate the between study variance component (τ2) within each subgroup?

5. RE models: we apply two weights for the overall pooled mean effect (and for subgroup means)
What does this tell us?
Combine within study and
between study variance
Weighted mean is given by
Note that τ2 can differ by subgroup
wi = 1 𝑉 𝑖
*wi= 1 𝑉 𝑖 + 𝜏 2 ,where 𝑣 𝑖 = 𝑠𝑒 2
M*= 𝑖=1 𝑘 ∗𝑊 𝑖 ∗𝑑 𝑖 𝑖=1 𝑘 ∗ 𝑊 𝑖

6. I2 statistic – what is it, why is it important?
“What proportion of the observed variance reflects real differences in effect size?” (Borenstein, p. 117)
I2 = percentage of variation attributable to heterogeneity (true study effects); easy computation:
I2 = (Q − df) Q X 100
Q = observed WSS, df = expected WSS
Q – df = excess variation
I2 lies between 0% and 100%; 0% = no heterogeneity
Benchmarks: (1) low = 25-50%, (2) moderate = 50-75%, (3) high >= 75%

7. How do we derive estimates of τ2 and I2 (Borenstein, p. 111)

8. Borenstein et al. tutoring example
In the chapter 19 reading, Borenstein et al. use an example of tutoring interventions (group A and group B) for their heterogeneity example.
On p. 165, Borenstein et al. report τ2 (between study variance) for group A and group B. How do they derive these estimates? Check Table 19.2 on p Take 5 minutes to discuss in pairs

9. STATA Activity Please download the Stata files: RER2013.DTA
Create a do file and run the following commands
metan es se, fixed lcols (author)
metan es se, random lcols (author)
metan es se, fixed second (random) lcols (author)
If you want to suppress the figures, use this command
metan es se, fixed nograph lcols (author)
metan es se, random nograph lcols (author)

10. Compare and contrast (use STATA results to complete this table)
Key results
Fixed effect model
Random effects model
1. Combined M (pooled ES)
2. 95% CI around M
3. Z test and associated p
5. Q total
6. I2
7. Estimate of τ2
n/a
Why does the combined M, the 95% CI around the combined M, and z tests and associated p-value differ in the 2 models?
How did STATA compute Q total?
How did STATA compute I2?
How did STATA compute τ2?

11. Define mixed effect model
Borenstein, p. 183 – “Mixed effect model”

12. I changed a few details for the STATA file for the class demo, but the mixed effects idea is key.
Kim and Quinn (2013) – Use random effects to estimate mean weighted effect size, overall and for each subgroup
And assume that each “summer intervention type” is fixed (only 1 of 2 types of treatments; anyone who does this analysis would use the same subgroups; thus, the subgroups are not randomly sampled from a larger population)

13. Fixed Effect: Subgroup Replication in Excel
(2a) PRACTICE-FE-Subgroup
Complete grid in yellow including subgroup estimates for:
Key results
1. Combined M (pooled ES)
2. 95% CI around M
3. Z test and associated p
4. Estimate of τ2
Compare with STATA FE subgroup results
Do you get similar results in Excel as you did in STATA

14. Random Effects: Subgroup Replication in Excel
(3a)PRACTICE-RE-Subgroup
Everyone completes yellow highlights in teams.
Each team reports back on:
Team 1 – Column W and X (overall mean weighted M, 95% CI, z test and p-value)
Team 2 – Column AC (compare subgroup means)
Team 3 – Column AH to AJ
ANSWER tabs (scaffold if you get lost and need help with formulas)

15. Team 1: Column W and X
How do we compute the weighted mean overall, and for classroom and home, and the 95% CI?

16. Team 2: Column AC
What is the difference in subgroup means, and the 95% CI around this difference?

17. Team 3 – Column AH to AJ

18. How would we write up our results and communicate the key results:
The grand (total) mean was .11 [CI = .047, 0.179], which was significantly different from 0. The Q total of was significant (p < .01) and the I2 value of 65% indicated moderate heterogeneity among studies and suggested that there were one or more study factors explaining variability in effects. In order to understand which type of summer reading intervention was most effective, we conducted a moderator analysis to compare the mean effects for classroom and home-based interventions.
A non-significant Qb(1) statistic of 0.017, indicated that the mean effect sizes were statistically equivalent across the two types of summer reading interventions.
Follow-up z-tests indicated that there was no statistically significant difference in the average weighted effect size of .12, 95% CI [.03, .21] for classroom interventions and the average weighted effect size of .11, 95% CI [.01, .21] for home interventions.

19. 5-7 Minute break
Please juxtapose Durlak et al. (2011) and Kim and Quinn (2013)

20. How I used my model paper as an outline
Readings: Durlak et al. (2011) and Kim and Quinn (2013)
Questions in pairs: Create 2 columns and find the parallel sub-heading / table in Kim and Quinn that mirrors Durlak et al. (2011). Lets review together.
Durlak et al. (2011)
Kim & Quinn (2013)
What is social and emotional learning?
Defining summer reading interventions
What is known about the impact…
Recent relevant reviews
Research hypotheses and study goals
Current meta-analysis: RQ and hypotheses

21. Peer Review and Meta-analysis
Readings: RER peer review comments and first draft – I want you to get a sense of how reviewers review meta-analysis. Try to bucket the criticism into these categories
1. hypothesis formulation (precise and relevant hypothesis)
2. retrieval and coding of studies
3. analysis of study results and characteristics
4. interpretation of meta-analytic outcomes
MARS – APA standards – reviewers use this checklist

22. Exercises and Independent Study
Exercise 3 – directions
Can work with a partner. Due October 8:30am. Will be posted by this Saturday at 3pm. Questions next week in class.
Independent Study  Presentation  Final Paper (1-1 meeting)
On (M) Oct. 5pm, you will post your independent study form on canvas and then I’ll use this to guide our 1-1 meeting.
Module 2 – syllabus updates
Oct. 18 – combined 2 weeks, and will continue this topic through Nov. 1
Nov. 1, 8, 15 – cover advanced topics in Borenstein readings
Nov. 15 – will assign required readings to cover special topics addressing student needs
Nov. 23 – student presentations (I said minutes in syllabus, but will add time for Q/A so more like 15-18)
Nov – final 1-1 meeting to discuss project goals following S32 and ways I can support your work