(1.7) Modeling Heterogeneity; Interpreting evidence (Part II) Professor James S. Kim S32-Accumulating Evidence

Agenda Part I Part II – Activity Review: modeling heterogeneity in meta-analysis Conduct moderator analysis of Kim & Quinn (2013) using random effects models Replication in STATA and Excel Part II – Activity Readings: Durlak et al. paper and codebook and APA-MARS standards – outline and model for my paper (I’ll show you how we used our model to organize our outline) Exercise 3 – directions and goals Independent Study  Presentation  Final Paper (1-1 meeting) Module 2 – syllabus updates

Goals Understand how to generate separate estimates of τ2 (between study variance component) when pooling means effects by subgroup (NB – use fixed effect models to generate the estimates) Replicate STATA results in Excel Be able to use your “model” meta-analysis to outline your own study (I will use Durlak et al. 2011)

Modeling heterogeneity in meta-analysis Three core questions plus 1 more question (1) Is there a common, fixed effect? Is there evidence of heterogeneity in true effects(Q) (2) If we reject the null hypothesis of homogeneous effects, how much do the mean effects vary across studies—i.e., what is the between study variance? (τ2) (3) If we reject the null hypothesis of homogeneous effects, what’s the true study-to-study heterogeneity in effects? (I2) (4) If we use a random effects model to pool mean effects by subgroup, how do we estimate the between study variance component (τ2) within each subgroup?

RE models: we apply two weights for the overall pooled mean effect (and for subgroup means) What does this tell us? Combine within study and between study variance Weighted mean is given by Note that τ2 can differ by subgroup wi = 1 𝑉 𝑖 *wi= 1 𝑉 𝑖 + 𝜏 2 ,where 𝑣 𝑖 = 𝑠𝑒 2 M*= 𝑖=1 𝑘 ∗𝑊 𝑖 ∗𝑑 𝑖 𝑖=1 𝑘 ∗ 𝑊 𝑖

I2 statistic – what is it, why is it important? “What proportion of the observed variance reflects real differences in effect size?” (Borenstein, p. 117) I2 = percentage of variation attributable to heterogeneity (true study effects); easy computation: I2 = (Q − df) Q X 100 Q = observed WSS, df = expected WSS Q – df = excess variation I2 lies between 0% and 100%; 0% = no heterogeneity Benchmarks: (1) low = 25-50%, (2) moderate = 50-75%, (3) high >= 75%

How do we derive estimates of τ2 and I2 (Borenstein, p. 111)

Borenstein et al. tutoring example In the chapter 19 reading, Borenstein et al. use an example of tutoring interventions (group A and group B) for their heterogeneity example. On p. 165, Borenstein et al. report τ2 (between study variance) for group A and group B. How do they derive these estimates? Check Table 19.2 on p. 155. Take 5 minutes to discuss in pairs

STATA Activity Please download the Stata files: RER2013.DTA Create a do file and run the following commands metan es se, fixed lcols (author) metan es se, random lcols (author) metan es se, fixed second (random) lcols (author) If you want to suppress the figures, use this command metan es se, fixed nograph lcols (author) metan es se, random nograph lcols (author)

Compare and contrast (use STATA results to complete this table) Key results Fixed effect model Random effects model 1. Combined M (pooled ES) 2. 95% CI around M 3. Z test and associated p 5. Q total 6. I2 7. Estimate of τ2 n/a Why does the combined M, the 95% CI around the combined M, and z tests and associated p-value differ in the 2 models? How did STATA compute Q total? How did STATA compute I2? How did STATA compute τ2?

Define mixed effect model Borenstein, p. 183 – “Mixed effect model”

I changed a few details for the STATA file for the class demo, but the mixed effects idea is key. Kim and Quinn (2013) – Use random effects to estimate mean weighted effect size, overall and for each subgroup And assume that each “summer intervention type” is fixed (only 1 of 2 types of treatments; anyone who does this analysis would use the same subgroups; thus, the subgroups are not randomly sampled from a larger population)

Fixed Effect: Subgroup Replication in Excel (2a) PRACTICE-FE-Subgroup Complete grid in yellow including subgroup estimates for: Key results 1. Combined M (pooled ES) 2. 95% CI around M 3. Z test and associated p 4. Estimate of τ2 Compare with STATA FE subgroup results Do you get similar results in Excel as you did in STATA

Random Effects: Subgroup Replication in Excel (3a)PRACTICE-RE-Subgroup Everyone completes yellow highlights in teams. Each team reports back on: Team 1 – Column W and X (overall mean weighted M, 95% CI, z test and p-value) Team 2 – Column AC (compare subgroup means) Team 3 – Column AH to AJ ANSWER tabs (scaffold if you get lost and need help with formulas)

Team 1: Column W and X How do we compute the weighted mean overall, and for classroom and home, and the 95% CI?

Team 2: Column AC What is the difference in subgroup means, and the 95% CI around this difference?

Team 3 – Column AH to AJ

How would we write up our results and communicate the key results: The grand (total) mean was .11 [CI = .047, 0.179], which was significantly different from 0. The Q total of 94.94 was significant (p < .01) and the I2 value of 65% indicated moderate heterogeneity among studies and suggested that there were one or more study factors explaining variability in effects. In order to understand which type of summer reading intervention was most effective, we conducted a moderator analysis to compare the mean effects for classroom and home-based interventions. A non-significant Qb(1) statistic of 0.017, indicated that the mean effect sizes were statistically equivalent across the two types of summer reading interventions. Follow-up z-tests indicated that there was no statistically significant difference in the average weighted effect size of .12, 95% CI [.03, .21] for classroom interventions and the average weighted effect size of .11, 95% CI [.01, .21] for home interventions.

5-7 Minute break Please juxtapose Durlak et al. (2011) and Kim and Quinn (2013)

How I used my model paper as an outline Readings: Durlak et al. (2011) and Kim and Quinn (2013) Questions in pairs: Create 2 columns and find the parallel sub-heading / table in Kim and Quinn that mirrors Durlak et al. (2011). Lets review together. Durlak et al. (2011) Kim & Quinn (2013) What is social and emotional learning? Defining summer reading interventions What is known about the impact… Recent relevant reviews Research hypotheses and study goals Current meta-analysis: RQ and hypotheses

Peer Review and Meta-analysis Readings: RER peer review comments and first draft – I want you to get a sense of how reviewers review meta-analysis. Try to bucket the criticism into these categories 1. hypothesis formulation (precise and relevant hypothesis) 2. retrieval and coding of studies 3. analysis of study results and characteristics 4. interpretation of meta-analytic outcomes MARS – APA standards – reviewers use this checklist

Exercises and Independent Study Exercise 3 – directions Can work with a partner. Due October 25 @ 8:30am. Will be posted by this Saturday at 3pm. Questions next week in class. Independent Study  Presentation  Final Paper (1-1 meeting) On (M) Oct. 24 @ 5pm, you will post your independent study form on canvas and then I’ll use this to guide our 1-1 meeting. Module 2 – syllabus updates Oct. 18 – combined 2 weeks, and will continue this topic through Nov. 1 Nov. 1, 8, 15 – cover advanced topics in Borenstein readings Nov. 15 – will assign required readings to cover special topics addressing student needs Nov. 23 – student presentations (I said 10-12 minutes in syllabus, but will add time for Q/A so more like 15-18) Nov. 29-30 – final 1-1 meeting to discuss project goals following S32 and ways I can support your work