1. Using and Reporting Measures of Effect Size Roger E. Kirk Department of Psychology & Neuroscience Baylor University

2. Three Categories of Measures of Effect Magnitude 1. Measures of effect size (typically, standardized mean differences) 2. Measures of strength of association 3. A large category of other kinds of measures 2

3. 1. Estimate the sample size required to achieve an acceptable power 2. Integrate the results of empirical research studies in meta-analyses 3. Supplement the information provided by null hypothesis significance tests 4. Determine whether research results are practically significant Four Purposes of Measures of Effect Magnitude 3

4. 1. Answers the wrong question What we want to know is the probability that the null hypothesis is true, given our data: Null hypothesis significance testing tells us the probability of obtaining our data or more extreme data if the null hypothesis is true: Four Criticisms of Null Hypothesis Significance Testing 4

5. 2. Is a trivial exercise According to John Tukey “the effects of A and B are always different—in some decimal place—for any A and B. Thus asking ‘Are the effects different?’ is foolish.” Four Criticisms of Null Hypothesis Significance Testing (continued) 5

6. According to Bruce Thompson “Statistical testing becomes a tautological search for enough participants to achieve statistical significance. If we fail to reject, it is only because we have been too lazy to drag in enough participants” Four Criticisms of Null Hypothesis Significance Testing (continued) 6

7. 3. Requires us to make a dichotomous decision from a continuum of uncertainty The adoption of.05 as as the dividing point between significance and non-significance is quite arbitrary. Four Criticisms of Null Hypothesis Significance Testing (continued) 7

8. 4. Does not address the question of whether results are important, valuable, or useful: that is, their practical significance. Four Criticisms of Null Hypothesis Significance Testing (continued) 8

9. 1.Is an observed effect real or should it be attributed to chance? 2.If the effect is real, how large is it? 3.Is the effect large enough to be useful? Three Basic Questions that Researchers Want to Answer from Their Research 9

10. “Because confidence intervals combine information on location and precision and can often be used to infer significance levels, they are, in general the best reporting strategy... Multiple degree-of-freedom indicators are often less useful than effect-size indicators that decompose multiple degree-of-freedom tests into one degree-of-freedom effects... Recommendation of the APA Publication Manual 10

11. (1)Cohen’s Three ways to estimate Cohen Glass Hedges Effect size 11

12.  = 0.2 is a small effect = 0.5 is a medium effect = 0.8 is a large effect Guidelines for Interpreting 12

13. (1) Sample estimators of omega squared and the intraclass correlation Strength of Association 13

14. =.001 is a small association =.059 is a medium association =.138 is a large association Guidelines for Interpreting Omega Squared 14

15. Effect SizeStrength of AssociationOther Measures Cohen d, f, g, h, q, Glass g ’ Hedges g Mahalanobis D Mean 1 – Mean 2 Mdn 1 – Mdn 2 Mode 1 – Mode 2 Rosenthal Tang Thompson d* Wilcox Measures of Effect Magnitude ________________________________________________________________ r, r pb, r 2, R, R 2, ,  2 Chamber r e Cohen f  Contingency coef C Cramér V Fisher Z Friedman r m Goodman Herzberg R 2 Kelly _______________________________________________________________ Abs. risk reduction ARR Cliff p Cohen U 1, U 2, U 3 Shift function Dunlap CL R Grisson PS Logit d ’ McGraw & Wong CL Odds ratio Preece ratio of success Probit d ’ Relative risk RR Sánchez-Meca d Cox 15

16. Effect SizeStrength of AssociationOther Measures Wilcox & Muska More Measures of Effect Magnitude ________________________________________________________________ Kendall W Lord R 2 Olejnik r equivalent r alerting r contrast r effect size Tatsuoka Wherry R 2  _______________________________________________________________ Rosenthal & Rubin BESD Rosenthal & Rubin ES counter null Wilcoxon 16

17. (1) (2) Two Ways to Estimate the Denominator of Cohen’s 17

18. Effect of the Unreliability of the Dependent Variable, Y, On the Proportion of Explained Variance 18

19. Aspirin groupp A =.01259 Placebo groupp P =.02166 p A – p P =.01259 –.02166 = –.009 Double-Blind Study of 22,071 Men Physicians 19

20. THE END 20