1. Interpreting Null Hypothesis Significance Tests (NHSTs) Joe Stevens, Ph.D. Educational Psychology 120 Simpson Hall 277-4203, jstevens@unm.edujstevens@unm.edu Presentation available at: http://www.unm.edu/~jstevens/papers/nhst.ppt © Stevens, 2002

2. Which study is “stronger”? Study A: t (398) = 2.30, p =.022 Study B: t (88) = 2.30, p =.024 Examples inspired by Rosenthal & Gaito (1963).

3. Answer: Study B ω 2 for Study A =.0106 ω 2 for Study B =.0455

4. Which study is “stronger”? Study P: F(2, 697) = 5.30, p =.00519 Study Q: F(2, 97) = 5.30, p =.00654

5. Answer: Study Q  2 for Study P =.0150  2 for Study Q =.0985

6. Which is “significant”/important? Study T: F = 63.62, p <.0000001 Study U: F = 5.40, p =.0486

7.  2 for Study T =.01, N = 6,300  2 for Study U =.40, N = 10 Study T is a very small effect, Study U is a much larger effect Significance”/importance cannot be determined statistically

8. Significance Test Results = Effect Size X Size of Study t = X

9. Significance Test Results = Effect Size X Size of Study t = X

10. Significance Test Results = Effect Size X Size of Study t = d X

11. Significance Test Results = Effect Size X Size of Study F = Xdf

12. Significance Test Results = Effect Size X Size of Study F = X

13. Interpreting Statistical Results NHST results (t, F, p, etc.) answer the question: Is there a relationship between variables (yes/no)? Effect size (d, g) and strength of association measures (r 2, ω 2, η 2 ) answer the question: How strong is the relationship?

14. NHST results should always be accompanied by estimation of effect size; report both Estimation of statistical power is also advisable, especially when NHST results are not statistically significant Use of Confidence Intervals can also aid proper interpretation of NHST results

15. Bibliography Cohen, J. (1994). The earth is round (p <.05). American Psychologist, 49, 997– 1003. Harlow, L. L. Mulaik, S. A., & Steiger, J. H. (1997). What if there were no significance tests? Hillsdale, NJ: Erlbaum. Rosenthal & Gaito (1963). The interpretation of levels of significance by psychological researchers. Journal of Psychology, 55, 33-38. Rosenthal, R. & Rosnow, R. L. (1991). Essentials of behavioral research (2 nd Ed.). New York: McGraw-Hill, Inc. Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25 (2), 26– 30. Wilkinson, L. & Task Force on Statistical Inference (1999). Statistical Methods in Psychology Journals: Guidelines and Explanations, American Psychologist, 54 (8), 594–604. [available at: http://www.apa.org/journals/amp/amp548594.html#c13].