1. Lecture 5 EPSY 642 Victor Willson Fall 2009

2. EFFECT SIZE DISTRIBUTION Hypothesis: All effects come from the same distribution What does this look like for studies with different sample sizes? Funnel plot- originally used to detect bias, can show what the confidence interval around a given mean effect size looks like Note: it is NOT smooth, since CI depends on both sample sizes AND the effect size magnitude

3. EFFECT SIZE DISTRIBUTION Each mean effect SE can be computed from SE = 1/  (w) For our 4 effects: 1: 0.200525 2: 0.373633 3: 0.256502 4: 0.286355 These are used to construct a 95% confidence interval around each effect

4. EFFECT SIZE DISTRIBUTION- SE of Overall Mean Overall mean effect SE can be computed from SE = 1/  (  w) For our effect mean of 0.8054, SE = 0.1297 Thus, a 95% CI is approximately (.54, 1.07) The funnel plot can be constructed by constructing a SE for each sample size pair around the overall mean- this is how the figure below was constructed in SPSS, along with each article effect mean and its CI

6. EFFECT SIZE DISTRIBUTION- Statistical test Hypothesis: All effects come from the same distribution: Q-test Q is a chi-square statistic based on the variation of the effects around the mean effect Q =  w i ( g – g mean ) 2 Q  2 (k-1) k

7. Example Computing Q Excel file effectdw Qiprob(Qi)sig? 10.585.43 0.71515980.397736175no 2-0.0510.24 0.73262480.392033721no 30.524.35 0.39579490.52926895no 40.029.69 0.3663190.545017585no 5-0.3040.65 10.6973490.001072891yes 60.1429.94 0.16866160.681304025no 70.6854.85 11.7274520.000615849yes 8-0.024.00 0.21256220.644766516no 0.2154 Q=25.015924 df7 prob(Q)=0.0007539

8. Computational Excel file Open excel file: Computing Q Enter the effects for the 4 studies, w for each study (you can delete the extra lines or add new ones by inserting as needed) from the Computing mean effect excel file What Q do you get? Q = 39.57 df=3 p<.001

9. Interpreting Q Nonsignificant Q means all effects could have come from the same distribution with a common mean Significant Q means one or more effects or a linear combination of effects came from two different (or more) distributions Effect component Q-statistic gives evidence for variation from the mean hypothesized effect

10. Interpreting Q- nonsignificant Some theorists state you should stop- incorrect. Homogeneity of overall distribution does not imply homogeneity with respect to hypotheses regarding mediators or moderators Example- homogeneous means correlate perfectly with year of publication (ie. r= 1.0, p<.001)

11. Interpreting Q- significant Significance means there may be relationships with hypothesized mediators or moderators Funnel plot and effect Q-statistics can give evidence for nonconforming effects that may or may not have characteristics you selected and coded for

12. MEDIATORS Mediation: effect of an intervening variable that changes the relationship between an independent and dependent variable, either removing it or (typically) reducing it. Path model conceptualization: TreatmentOutcome Mediator

13. MEDIATORS Statistical treatment typically requires both paths ‘a’ and ‘b’ to be significant to qualify as a mediator. Meta- analysis seems not to have investigated path ‘a’ but referred to continuous predictors as regressors Lipsey and Wilson(2001) refer to this as “Weighted Regression Analysis” TreatmentOutcome Mediator a b

14. Weighted Regression Analysis Model: e = b X + residual Regression analog: Q = Q regression + Q residual Analyze as “weighted least squares” in programs such as SPSS or SAS In SPSS the weight function w is a variable used as the weighting

15. Weighted Regression Analysis Emphasis on predictor and its standard error: the usual regression standard error is incorrect, needs to be corrected (Hedges & Olkin, 1985): SE’ b = SE b / (MSe) ½ where SE b is the standard error reported in SPSS, and MSe is the reported regression mean square error

16. Weighted Regression Q-statistics Q regression = Sum of Squares regression df = 1 for single predictor Q residual = Sum of Squares residual df = # studies - 2 Significance tests: Each is a chi square test with appropriate degrees of freedom

17. 98.999.0531191.702628.781319 712.810.0324260.267212.369946 89.0911.5630200.7561211.22962 810.8610.5225250.6532111.867084 97.737.8622281.541429.530347 610.1110.7724260.3507112.291338 78.576.9134160.4438110.651743 79.598.5322281.1245210.659409 87.9810.9230200.542111.591384 912.698.1628220.6337111.739213 58.6110.572822-0.5976211.80079 59.347.4324260.3771112.262378 710.3910.126240.7234211.714913 68.669.421290.2413112.094229 79.169.0425250.6637111.847643 88.186.4330200.9038210.928737 910.049.8425250.4603112.177485 712.3311.422280.3948112.087879 78.8310.672327-0.1726212.374215 810.888.8126240.4633112.154409 89.58.0928220.8481211.317137 910.4210.1218320.7114110.885366 911.827.1328220.5407111.891682 611.698.1123270.4926112.05664

18. ANOVA b,c ModelSum of SquaresdfMean SquareFSig. Regression19.166119.16612.096.002 a Residual34.858221.584 Total54.02423 a.Predictors: (Constant), AGE b.Dependent Variable: HEDGE d* c. Weighted Least Squares Regression - Weighted by w Coefficients a,b Model Unstandardized CoefficientsStandardized Coefficients tSig. BStd. ErrorBeta (Constant)-1.037.465-2.230.036 AGE.215.062.5963.478.002 a. Dependent Variable: HEDGE d* b. Weighted Least Squares Regression - Weighted by w SPSS ANALYSIS OUTPUT

19. Example See SPSS “sample meta data set.sav” or the excel version “sample meta data set regression” The d effect is regressed on Age b = 0.215, SE b = 0.062, MSe = 1.584 Thus, SE’ b = 0.062 / (1.584) ½ = 0.0493 A 95% CI around b gives (0.117, 0.313) for the regression weight of age on outcome, p<.001

20. Q-statistic tests Q regression = 19.166 with df=1, p <.001 Q residual = 34.858 with df=22, p =.040 So- are the residuals homogeneous or not? Given a large number of significance tests, one might require the Type I error rate for such tests to be.001 or something small