1. Effect Sizes for Continuous Variables William R. Shadish University of California, Merced

2. Indices for Treatment Outcome Studies Correlation coefficient (r) between treatment and outcome Standardized mean difference statistic (d) Either can be transformed into the other, so we will work with d since it is most common. Other indices do exist but are rare in social science meta-analyes.

3. Estimating d d itself Algebraic equivalents to d Good approximations to d Methods that require intraclass correlation Methods that require ICC and change scores Methods that underestimate effect Note: Italicized methods will be covered in this workshop.

4. Sample Data Set I: Two Independent Groups TreatmentComparison 32 424 45566 67 78 79 Mean5.25.4 Standard Deviation1.3982.319 Sample Size1010 Correlation between treatment and outcome is r = -.055

5. Calculating d

6. Algebraic Equivalent: Between Groups t-test on raw posttest scores,

7. Algebraic Equivalent: t-test for two matched groups, sample sizes, correlation between groups

8. Algebraic Equivalent: Two-group between-groups F-statistic on raw posttest scores (Data Set I)

9. Algebraic Equivalent: Multifactor Between Subjects ANOVA with Two Treatment Conditions 1.Sums of Squares and Degrees of Freedom for all sources, and Marginal Means for Treatment Conditions 2.Mean Squares and Degrees of Freedom for all sources, and Marginal Means for Treatment Conditions 3.Sums of Squares and Degrees of Freedom for all sources, with Cell Means and Cell Sample Sizes 4.Mean Squares and Degrees of Freedom for all sources, with Cell Means and Cell Sample Sizes 5.Cell means, cell sample sizes, the F-statistic for the treatment factor, and the degrees of freedom for the error term 6.F-statistics and degrees of freedom for all sources, sample size for treatment and comparison groups, where treatment factor has only two levels

10. Example: Sums of Squares and Degrees of Freedom for all sources, and Marginal Means for Treatment Conditions: Data Set II B1B2B3 A18108 486 064 A214415 10212 609 Row B1B2B3 Marginal A14.08.06.06.0 (3)(3)(3)(9) A210.02.012.08.0 (3)(3)(3)(9) Column7.58.05.57.0 Marginal (6)(6)(6)(18) Sum of Squaresdf Mean Square F Probability A18.000118.0002.038.179 B48.000224.0002.717.106 AB144.000272.0008.151.006 Residual106.000128.833 Total 316.00017 18.588

11. Example: Sums of Squares and Degrees of Freedom for all sources, and Marginal Means for Treatment Conditions For a two group one factor ANOVA: For a two factor ANOVA: Which is the same as would have been obtained had Factor B not existed (with equal n per cell),

12. Algebraic Equivalent : Oneway two-group ANCOVA: Covariance error term, F for covariate, raw score means, and total sample size (Data Set III) Time 1Time 2Change Time 2 Group Mean.00.00.00 Group 13.001.00-2.00 1.333 4.003.00-1.00 4.002.00-2.00 Group 25.004.00-1.00 3.667 7.005.00-2.00

13. ANCOVA Table, Time 2 as Outcome, Time 1 as covariate Source Sum of SquaresdfMean SquareFSig. Covariate7.50017.50012.273.039 Groups.0051.005.008.932 Error1.8333.611 Total17.50053.500 Note: This table was computed using the unique sum of squares method as defined in SPSS for Windows Version 7.5.

14. Algebraic Equivalent: Oneway two-group ANCOVA: Covariance error term, F for covariate, raw score means, and total sample size (Data Set III) Which is the same as would have been obtained had the standard method been applied to the Time 2 scores

15. Algebraic Equivalent: Exact Probability and Sample Sizes If exact p value from t-test or two group F-test Use sample size to get df, which in turn allows you to get exact t statistic Then apply t-test method previously shown From Data Set I – exact probability for t-test was p =.818. –for df = 20-2 = 18, t =.2336 –so d = -.104, same as before

16. Algebraic Equivalent: r to d To convert r to d uncorrected for small sample bias, using Data Set I: Which is the same as originally obtained using the standard formula for d

17. Algebraic Equivalent: Raw Data Sometimes raw data is tabled as, say, –Treatment group N = 10: A = 20%, B =20%, C = 30%, D = 20%, and F = 10% –Comparison group N = 10: A = 10%, B = 20%, C = 20%, D = 30%, and F = 20% Create raw data as, say, A = 4, B = 3, C = 2, D = 1, and F = 0 –treatment group is 4, 4, 3, 3, 2, 2, 2, 1, 1, 0 –comparison group is 4, 3, 3, 2, 2, 1, 1, 1, 0, 0 Then d =.377

18. Good Approximation Three-group or higher between-groups oneway ANOVA on posttest scores: group means, sample sizes, and F-statistic,

19. Example Data Set IV PosttestGroup Mean.00 Group 11.001.333 3.00 2.00 Group 24.003.667 5.00 2.00 Group 31.001.333 1.00 G = 2.111 F = 3.267 This is similar but not identical to d = -1.53 using the standard method comparing groups 1 and 2. Difference due to different s p.

20. Good Approximation Three-group or higher between-groups oneway ANOVA on raw posttest scores: treatment and comparison group means and mean square error. For Data Set IV:

21. Good Approximations: Two-Factor RM-ANOVA (groups x time) Between-groups mean square error, within- groups mean square error, posttest means, and sample sizes F-ratio for groups, F-ratio for time, cell means and sample sizes F-ratio for groups, F-ratio for group × time interaction, cell means and sample sizes

22. Example: Data Set V This data set is taken from Winer (1972, p. 525). It presents a two-factor model with factor A as a between subjects factor having two levels, A1 and A2, and factor B as a within-subjects factor having four levels (columns B1 through B4). The raw data are: B1B2B3B4 A10053 3154 4362 A24278 5466 7589

23. Example: Data Set V RM-ANOVA Here are the cell means (and sample sizes) for the same data, along with marginals and grand means. Row B1B2B3B4Marginal A12.331.335.333.003.00 (3)(3)(3)(3)(12) A25.333.677.007.675.917 (3)(3)(3)(3)(12) Col3.832.506.175.334.458 = Grand Mean Marginal (6)(6)(6)(6)(24) Repeated Measures ANOVA Table Tests of Within-Subjects Effects SourceSum of Squares df Mean Square F Probability B 47.4583 15.81912.798.000 AB 7.4583 2.486 2.011.166 WS Error 14.83312 1.236 Tests of Between-Subjects Effects Source Sum of Squares df Mean Square F Probability A 51.0421 51.04211.893.026 BS Error 17.1674 4.292

24. Between-groups mean square error, within- groups mean square error, posttest means, and sample sizes: Data Set V Assuming Time 4 is the time point of interest (e.g., it is the posttest, or the followup), then:

25. Methods that underestimate effect size I Results reported as verbally “significant”, or as p <.05 or <.01 etc., with sample size Use previous method to convert p to t, and then use t to compute d as before. In Data Set I using p <.05, this method would yield t = -2.004, yielding d = -.939. Underestimates d because t will increase as p decreases, and p =.05 is too high. Be careful to distinguish 1 vs 2 tailed tests.

26. Methods that underestimate effect size II Results reported only as nonsignificant. Omitting them from the meta-analysis results in an overestimate of average d. A typical solution is to code them as d = 0 (introduces a constant variance problem), but then do sensitivity analyses. More sophisticated solutions exist such as maximum likelihood imputation.

27. Discussion Many more methods exist The standard error for all but d and its algebraic equivalents are typically unknown Whether to use the approximations or not involves the same tradeoffs as with results reported only as nonsignificant (missing effect sizes vs approximate results) When doing a meta-analysis, good practice is to code effect size calculation method, and then explore its effects on outcome.

28. Computer Programs Lipsey and Wilson’s excel macro (free at http://mason.gmu.edu/~dwilsonb/ma.html) ES program (purchase at http://www.assess.com/ES.html) For more meta-analytic software, see http://faculty.ucmerced.edu/wshadish/Meta- Analysis%20Links.htm.