H676 Week 7 – Effect sizes and other issues Mayra Issues related to effect size Part 7 of the TEXT Brian Further methods Part 8 of the TEXT Moving from “what works” to “what happens” Pettigrew, 2015

Mayra, you’re on TEXT Chapters 31-34 Effect sizes rather than p-values Issues related to effect size Effect sizes rather than p-values Simpson’s Paradox Generality of the basic inverse-variance method

Brian TEXT Chapters 35-38 Further Methods Meta-analysis methods based on direction and p-values Further methods for dichotomous data Psychometric meta-analysis

Meta-analysis methods based on direction and p-values What are the different objectives of a meta-analysis of effect sizes vs a meta-analysis of p-values?

Different objectives MA of effect sizes MA of p-values Addresses the magnitude of the effect MA of p-values Addresses the null-hypothesis Tells us only that the effect is probably not zero

Other options What are other options for MA when it is not possible to calculate effect-sizes? What are the pros and cons of each?

Other options – Vote counting Has no validity (recall chapter 28) The absence of a sig effect is not evidence that the effect is absent. Why? Gets even worse with more studies Why? Small p could be due to small sample or lack of power

Other options – The sign test Simply count N of studies with findings in each direction This done without regard to statistical significance Therefore has little value! The statistical significance of the sign test can be tested using the sign test In Excel, the =BINOMDIST(n,N,0.5,TRUE) Where n=n of studies in smaller group, N=Total N of studies

Other options – Combining p-values Makes use of exact p-values Use only when sample sizes are not available If sample size is available, can back calculate an ES Convert 2-tailed tests to 1-tailed 1-tailed p-value includes information on direction as well as magnitude Run MA of the 1-tailed p-values Two possible statistical tests of the ultimate ES They test the null hypothesis that the effect is zero in all of the studies

Tests of sig. of analysis of 1-tailed p-values Fisher’s method: X2 = minus 2 times the sum of the logged p-values Distributed as X2 with degrees of freedom = 2*k Stouffer’s method: ZStouffer = sum of Zi divided by square root of k Follows a normal distribution like any other Z

Further methods for dichotomous data Two other methods for MA of odds ratios The Mantel-Haenzel method An alternative to the fixed-effect inverse-variance method Uses weighted average of ORs rather than of log ORs The one-step (Peto) method Is an inverse-variance method But uses an alternative approach to computing the odds ratio and variance for each study Offers some advantages when some studies have empty cells Both are available options in CMA

Psychometric meta-analysis Also known as: Validity generalization meta-analysis Hunter-Schmidt meta-analysis Two important issues: Artifact correction (adjusting for methodological limitations) of interest to anyone Psychometric MA – of interest when ESs are correlations

Psychometric meta-analysis Effect sizes are corrected for attenuation due to: Unreliability of measures Restricted range in either dependent or independent variables for any of the study samples Restricted variance when continuous variables are (artificially) dichotomized All corrections lead to larger ES and smaller variance In conventional MA, reliability would be examined in a meta-regression Some issues are sources of disagreement among uses of psychometric meta-analysis

And now, back to Mayra Moving from “what works” to “what happens” Pettigrew, 2015