1. Exercise 2-Effect Size Coding
Directions: complete this exercise on your computer and upload your answers to canvas. You should use borenstein chapter 4 and the formulas that I’ve given you to answer questions Note that question 7 asks about coding a particular study from your model meta-analysis.
Name:
Acknowledge Collaborators:

2. Cohen’s d Example from Kim & Quinn(2013): Schacter & Jo (2005)
Discovered through electronic search

3. Example: Schacter & Jo (2005) 𝑆 𝑤𝑖𝑡ℎ𝑖𝑛 =
Question (1) Compute Swithin and d. Note that the parentheses below are the standard deviations.
Example: Schacter & Jo (2005)
𝑆 𝑤𝑖𝑡ℎ𝑖𝑛 =
If only the overall sample size is reported for an experiment, assume treatment/control sample sizes are equal
d =

4. Question (2) Compute 𝑉 𝑑 and 𝑆 𝐸 𝑑
Example: Schacter & Jo (2005)
V d =
𝑆 𝐸 𝑑 =

5. Question (3) Compute J and Hedge’s g
Example: Schacter & Jo (2005) 𝐽= 𝑔=

6. Covariate-adjusted Cohen’s d
Sometimes a study might report a covariate-adjusted effect instead of unadjusted descriptive statistics
𝑑 𝑎𝑑𝑗 = 𝑌 1 𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑 − 𝑌 2 𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑 𝑆 𝑤𝑖𝑡ℎ𝑖 𝑛 𝑎𝑑𝑗 ,
where
𝑆 𝑤𝑖𝑡ℎ𝑖 𝑛 𝑎𝑑𝑗 = 𝑆 𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑 1− 𝑅 2 ,
where 𝑆 𝑎𝑑𝑗𝑢𝑠𝑡𝑒𝑑 is the covariate adjusted standard deviation
(Borenstein, 2009)
Q: does the std coeff SE in stata take into account the additional error from estimating the sd, or does it treat the sd as a constant?

7. Covariate-adjusted Cohen’s d
Alternatively, you can estimate 𝑑 𝑎𝑑𝑗 with the treatment coefficient in a regression table if the outcome used in the regression has been standardized
In a randomized trial, covariates are added to improve the precision of the treatment effect estimate, not to reduce bias. In an RCT, d and 𝑑 𝑎𝑑𝑗 should be similar (though the standard errors will not be).
Q: will the reg coefficient be exactly the same as the formula in the previous slide?

8. SE for Covariate-adjusted Cohen’s d
Remember that the SE for an ES must account for sampling variability not only in the means, but also in the pooled sd.
𝑉 𝑑 𝑎𝑑𝑗 = 𝑛 1 + 𝑛 2 1− 𝑅 2 𝑛 1 𝑛 𝑑 2 2( 𝑛 1 + 𝑛 2 )
If you are taking 𝑑 𝑎𝑑𝑗 from a regression coefficient, the SE on that coefficient will underestimate 𝑉 𝑑 𝑎𝑑𝑗 because it does not account for sampling variability in the sd.
Q: I believe the last point is correct?

9. Question (4) Compute SE for Covariate-adjusted Cohen’s d from Kim (2006)
Example from Kim (2006)
“discovered” through electronic search
𝑆𝐸 𝑑 𝑎𝑑𝑗 =
In practice with a large sample, the SE for d_adj will be similar to that of the reg coeff?

10. Question (5) Compute Covariate-adjusted Hedge’s g from Kim (2006)
Use the same Hedge’s g formula as before, with an adjustment to J for the loss of additional degrees of freedom by the inclusion of covariates
where 𝑑𝑓= 𝑛 1 + 𝑛 2 −2 −𝑞,
where q is the number of covariates in the model.
Hedge’s g =

11. Recovering an ES using the t-statistic
If a study reports just a t-statistic without means/standard deviations, the ES can be estimated using the following formula:
𝐸𝑆=𝑡 𝑛 1 + 𝑛 2 𝑛 1 𝑛 2

12. Question (6) Compute an effect size using the t-statistic from Allington et al. (2010)
Example from Kim & Quinn (2013): Allington et al. (2010)
Discovered through electronic search
“Our first comparison tested the hypothesis that the FCAT performances of the treatment students would exceed those of the control group. A t-test found statistically significant differences (t = 2.434, df = 1,328, p = .015) in the performance of the treatment and control students on the FCAT administered after three consecutive summer book distributions.” (p. 420)
𝐸𝑆 (𝑢𝑠𝑖𝑛𝑔 𝑡ℎ𝑒 𝑡−𝑠𝑡𝑎𝑡𝑖𝑠𝑡𝑖𝑐 𝑎𝑏𝑜𝑣𝑒)= show the computations you made:
There weren’t any studies that reported just the t-statistic; Allington reports both t-stat and ES

13. 2. What questions arose as you did this exercise?
Question (7) Using 1 of your primary studies from your model meta-analysis, explain how you coded the effect size.
For the final question, choose one of the primary studies included in your model meta-analysis. Provide enough detail so that the second rater could replicate the effect size coding.
1. Compute an effect size from your primary study. How, exactly, did you compute the effect size. In detail, tell me the relevant text or numbers (and page number) you used to generate the effect size.
2. What questions arose as you did this exercise?
3. How long did it take you to code one effect size? Please note that everyone will have a very different study to code. I tried coding most all of the effect sizes and am familiar with most of the issues that will arise, but I want you to get a sense of why some studies are easy to code (while others are difficult).
In class next week, I will ask you to share your answers to 1-3 above.

14. Type your Answers to Q7 here (add more slides if you need space)

15. References
Allington, R. L., McGill-Franzen, A., Camilli, G., Williams, L., Graff, J., Zeig, J., & Nowak, R. (2010). Addressing summer reading setback among economically disadvantaged elementary students. Reading Psychology, 31, 411–427. doi: /
Borenstein, M., Hedges, L. V., Higgins, J. P. T., Rothstein, H. R. Introduction to Meta-Analysis. West Sussex, UK: John Wiley & Sons, Lt.
Jacob, B. A., & Lefgren, L. (2004). Remedial education and student achievement: A regression-discontinuity analysis. Review of Economics and Statistics, 68, 226–244. doi: /
Kim, J. S. (2006). Effects of a voluntary summer reading intervention on reading achievement: Results from a randomized field trial. Educational Evaluation and Policy Analysis, 28, 335–355. doi: /
Kim, J.S., & Quinn, D.M. (2013). The effects of summer reading on low-income children’s literacy achievement from kindergarten to grade 8: A meta-analysis of classroom and home interventions. Review of Educational Research , 83(3) DOI: /
Schacter, J., & Jo, B. (2005). Learning when school is not in session: A reading summer day-camp intervention to improve the achievement of exiting first-grade students who are economically disadvantaged. Journal of Research in Reading, 28, 158–169. doi: /j x