Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall Ch. 12: Statistical Significance, Effect Size, and Power Analysis

Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall Null Hypothesis Significance Testing (NHST) Is an observed difference between groups real or merely the result of chance? One examines the probability associated with the observed difference. “p value”

Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall How Is NHST Sometimes Misunderstood or Misused? p values are not indicative of the size or importance of the treatment effect. “Not significant” is not the same as “No Effect” The p value is influenced by the size of the effect as well as N.

Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall Two Types of Statistical Hypotheses NULL HYPOTHESIS: No difference between the means or the correlation is zero in the population. ALTERNATIVE HYPOTHESIS: Means do differ or population correlation is not zero in the population.

Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall Possible Decision Outcomes in Hypothesis Testing

How can one determine whether or not r is statistically significant? Is r = 0 in the population? How can one determine whether or not r is statistically significant? Is r = 0 in the population? H 0 : r = O in population H A : r ≠ O in population H 0 : r = O in population H A : r ≠ O in population Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall

Two-Tailed vs. One-Tailed Hypothesis Tests Use when not specifying direction of difference Use when specifying direction of difference (e.g., greater/less than)

Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall Options When Reporting p values “p <.05, two-tailed” but too imprecise “.01 < two-tailed p <.02” better, but can be improved upon State precise p value e.g., p =.024 (two-tailed)

Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall Binomial Effect-Size Display (BESD) Way of examining the practical importance of an obtained effect size. “Display” because converts “success rates” in the experimental and control groups into a 2 x 2 table. “Binomial” because the two variables are presented as being dichotomous.

Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall p = r =.034 Aspirin’s Effect on Heart Attacks

Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall ( r/2) 2 100( r/2) Binomial Effect-Size Display of r =.034

Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall The Problem in Reporting r 2 as the Effect Size r 2 also called the: “Coefficient of determination” “proportion of variation explained” Not recommended as measure of effect size because: Loss of directionality Likely to be misconstrued as being unimportant.

Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall r 2 versus r effect size

Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall Smith: N = 80, p<.05, r =.22 Jones: N = 20, p>.30, r =.22 Statistical Power Analysis Power is 1 – β, the probability of not making a Type II error

Constructing Confidence Intervals Rosnow, Beginning Behavioral Research, 5/e. Copyright 2005 by Prentice Hall