1. Statistics for the Social Sciences
Psychology 340
Spring 2010
Effect sizes & Statistical Power

2. Reminders Don’t forget to complete homework 4 for Feb 9 (Tues)
And Quiz 4 (Chapter 8) by 11PM Wed (Feb 10)
Exam 1 Feb 11 (Thurs)

3. Outline Error types revisited Effect size: Cohen’s d
Statistical Power Analysis

4. Performing your statistical test
There really isn’t an effect
Real world (‘truth’)
There really is
an effect
H0 is correct
H0 is wrong
Reject H0
Experimenter’s conclusions
Fail to Reject H0

5. Performing your statistical test
There really isn’t an effect
Real world (‘truth’)
There really is
an effect
H0 is correct
H0 is wrong

6. Performing your statistical test
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
Real world (‘truth’)
H0 is correct
H0 is wrong
So there is only one distribution
So there are two distributions
The original (null) distribution
The new (treatment) distribution
The original (null) distribution

7. Performing your statistical test
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
Real world (‘truth’)
H0 is correct
H0 is wrong
So there is only one distribution
So there are two distributions
The original (null) distribution
The new (treatment) distribution
The original (null) distribution

8. Effect Size
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
Hypothesis test tells us whether the observed difference is probably due to chance or not
H0 is wrong
So there are two distributions
Memory example experiment:
How do they compare to the general population of memory patients who have a distribution of memory errors that is Normal, μ = 60, σ = 8?
After the treatment they have an average score of = 55 memory errors.
The new (treatment) distribution
The original (null) distribution
n = 16
p = .0062
Whether we “reject H0” or “fail to reject H0” depends on our sample size.
n = 4
p = .1075
n = 25
p = .0009

9. Effect Size
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
Hypothesis test tells us whether the observed difference is probably due to chance or not
It does not tell us how big the difference is
H0 is wrong
So there are two distributions
The new (treatment) distribution
The original (null) distribution
Effect size tells us how much the two populations don’t overlap

10. Effect Size Figuring effect size
But this is tied to the particular units of measurement
Figuring effect size
The new (treatment) distribution
The original (null) distribution
Effect size tells us how much the two populations don’t overlap

11. Effect Size Standardized effect size Cohen’s d
Puts into neutral units for comparison (same logic as z-scores)
Cohen’s d
The new (treatment) distribution
The original (null) distribution
Effect size tells us how much the two populations don’t overlap

12. Effect Size Standardized effect size Cohen’s d
Puts into neutral units for comparison (same logic as z-scores)
Cohen’s d
The new (treatment) distribution
The original (null) distribution
Assumption: the standard deviation of both these populations is the same
Effect size tells us how much the two populations don’t overlap

13. Effect Size Effect size conventions small d = .2 medium d = .5
large d = .8
The new (treatment) distribution
The original (null) distribution
Effect size tells us how much the two populations don’t overlap

14. Effect Size: An Example
Memory example experiment:
We give a n = 16 memory patients a memory improvement treatment.
so we are expecting a medium sized effect
We expect a reduction of around 5 errors for those who get the treatment
The original (null) distribution
Treatment distribution
We know that the distribution of errors for the general population of memory patients has the following properties: μ = 60, σ = 8?
55 =
= 60

15. Effect Size: An Example
Changing the sample size did not change the effect size
Memory example experiment:
We give a n = 4 memory patients a memory improvement treatment.
so we are expecting a medium sized effect
We expect a reduction of around 5 errors for those who get the treatment
The original (null) distribution
Treatment distribution
We know that the distribution of errors for the general population of memory patients has the following properties: μ = 60, σ = 8?
55 =
= 60

16. Error types There really isn’t an effect Real world (‘truth’)
H0 is correct
H0 is wrong
I conclude that there is an effect
Reject H0
Experimenter’s conclusions
I can’t detect an effect
Fail to Reject H0

17. Error types Real world (‘truth’) H0 is correct H0 is wrong
Type I error (α): concluding that
there is a difference between groups
(“an effect”) when there really isn’t.
H0 is correct
H0 is wrong
Type I error
Reject H0
Experimenter’s conclusions
Type II error (β): concluding that
there isn’t an effect, when there really is.
Fail to Reject H0
Type II error

18. Statistical Power
The probability of making a Type II error is related to Statistical Power
Statistical Power: The probability that the study will produce a statistically significant results if the research hypothesis is true (there is an effect)
So how do we compute this?

19. Statistical Power Real world (‘truth’) α = 0.05
H0: is true (is no treatment effect)
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
The original (null) distribution
This is the transformed distribution of sample means
α = 0.05
Reject H0
Fail to reject H0

20. Statistical Power Real world (‘truth’) α = 0.05
H0: is false (is a treatment effect)
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
This is also based on the transformed distribution of sample means
The new (treatment) distribution
The original (null) distribution
α = 0.05
Fail to reject H0
Reject H0

21. Statistical Power Real world (‘truth’)
H0: is false (is a treatment effect)
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
The new (treatment) distribution
The original (null) distribution
β = probability of a Type II error
α = 0.05
Failing to
Reject H0, even
though there is
a treatment effect
Reject H0
Fail to reject H0

22. Statistical Power Real world (‘truth’)
H0: is false (is a treatment effect)
Real world (‘truth’)
H0 is correct
H0 is wrong
Type I error
Type II error
The new (treatment) distribution
The original (null) distribution
β = probability of a Type II error
α = 0.05
Power = 1 - β
Failing to
Reject H0, even
though there is
a treatment effect
Probability of
(correctly)
Rejecting H0
Reject H0
Fail to reject H0

23. Statistical Power Steps for figuring power
1) Gather the needed information: mean and standard deviation of the Null Population and the predicted mean of Treatment Population

24. Statistical Power Steps for figuring power
2) Determine the properties of the distribution of sample means for these
2.0
2.0
(a) What is your sample size?
(b) What is your standard error (assume equal population standard deviations)?

25. Statistical Power Steps for figuring power
2.0
Statistical Power
Steps for figuring power
2) Determine the properties of the distribution of sample means for these
α = 0.05
(c) What is your critical z-score?
Here it is: 1-tailed, α=0.05, so from the table we see that it is

26. Statistical Power Steps for figuring power
3) Figure the raw-score cutoff point on the comparison distribution to reject the null hypothesis
α = 0.05
Transform this z-score to a raw score
2.0

27. Statistical Power Steps for figuring power
4) Figure the Z score for this same point, but on the distribution of means for treatment Population
Remember to use the
properties of the
treatment population!
Transform this raw score back to a z-score, within the treatment distribution

28. Statistical Power Steps for figuring power
5) Use the normal curve table to figure the probability of getting a score more extreme than that Z score
β = probability of a Type II error
From the unit normal table: Z(0.855) =
Power = 1 - β
2.0
The probability of detecting this an effect of this size from these populations is 80%

29. Statistical Power Factors that affect Power: α-level Sample size n
Population standard deviation σ
Effect size d
1-tail vs. 2-tailed

30. Statistical Power Factors that affect Power: α-level β Power = 1 - β
Change from α = 0.05 to 0.01
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

31. Statistical Power Factors that affect Power: α-level β Power = 1 - β
Change from α = 0.05 to 0.01
α = 0.05
α = 0.01 β
Power = 1 - β
Reject H0
Fail to reject H0

32. Statistical Power Factors that affect Power: α-level β Power = 1 - β
Change from α = 0.05 to 0.01
α = 0.05
α = 0.01 β
Power = 1 - β
Reject H0
Fail to reject H0

33. Statistical Power Factors that affect Power: α-level β Power = 1 - β
Change from α = 0.05 to 0.01
α = 0.05
α = 0.01 β
Power = 1 - β
Reject H0
Fail to reject H0

34. Statistical Power Factors that affect Power: α-level β Power = 1 - β
Change from α = 0.05 to 0.01
α = 0.05
α = 0.01 β
Power = 1 - β
Reject H0
Fail to reject H0

35. Statistical Power Factors that affect Power: α-level β Power = 1 - β
So as the a level gets
smaller, so does the
Power of the test
Change from α = 0.05 to 0.01
α = 0.05
α = 0.01 β
Power = 1 - β
Reject H0
Fail to reject H0

36. Statistical Power Factors that affect Power: Sample size β
Recall that sample size is related to the spread of the distribution
Change from n = 25 to 100
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

37. Statistical Power Factors that affect Power: Sample size β
Change from n = 25 to 100
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

38. Statistical Power Factors that affect Power: Sample size β
Change from n = 25 to 100
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

39. Statistical Power Factors that affect Power: Sample size β
Change from n = 25 to 100
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

40. Statistical Power Factors that affect Power: Sample size β
As the sample gets bigger, the standard error gets smaller and the Power gets larger
Change from n = 25 to 100
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

41. Statistical Power Factors that affect Power:
Population standard deviation
Change from σ = 25 to 20
Recall that standard error is related to
the spread of the distribution
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

42. Statistical Power Factors that affect Power:
Population standard deviation
Change from σ = 25 to 20
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

43. Statistical Power Factors that affect Power:
Population standard deviation
Change from σ = 25 to 20
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

44. Statistical Power Factors that affect Power:
Population standard deviation
Change from σ = 25 to 20
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

45. Statistical Power Factors that affect Power:
Population standard deviation
Change from σ = 25 to 20
As the σ gets smaller, the standard error
gets smaller and the Power gets larger
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

46. Statistical Power Factors that affect Power: Effect size μtreatment
Compare a small effect (difference) to a big effect
μtreatment
μno treatment
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

47. Statistical Power Factors that affect Power: Effect size β
Compare a small effect (difference) to a big effect
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0
μtreatment
μno treatment

48. Statistical Power Factors that affect Power: Effect size β
Compare a small effect (difference) to a big effect
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

49. Statistical Power Factors that affect Power: Effect size β
Compare a small effect (difference) to a big effect
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

50. Statistical Power Factors that affect Power: Effect size β
Compare a small effect (difference) to a big effect
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

51. Statistical Power Factors that affect Power: Effect size β
Compare a small effect (difference) to a big effect
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

52. Statistical Power Factors that affect Power: Effect size β
Compare a small effect (difference) to a big effect
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

53. Statistical Power Factors that affect Power: Effect size β
Compare a small effect (difference) to a big effect
α = 0.05
As the effect gets bigger, the Power gets larger β
Power = 1 - β
Reject H0
Fail to reject H0

54. Statistical Power Factors that affect Power: 1-tail vs. 2-tailed β
Change from α = 0.05 two-tailed to α = 0.05 two-tailed
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

55. Statistical Power Factors that affect Power: 1-tail vs. 2-tailed β
Change from α = 0.05 two-tailed to α = 0.05 two-tailed
p = 0.025
α = 0.05 β
Power = 1 - β
Reject H0
Fail to reject H0

56. Statistical Power Factors that affect Power: 1-tail vs. 2-tailed β
Change from α = 0.05 two-tailed to α = 0.05 two-tailed
α = 0.05
p = 0.025 β
Power = 1 - β
p = 0.025
Reject H0
Fail to reject H0

57. Statistical Power Factors that affect Power: 1-tail vs. 2-tailed β
Change from α = 0.05 two-tailed to α = 0.05 two-tailed
α = 0.05
p = 0.025 β
Power = 1 - β
p = 0.025
Reject H0
Fail to reject H0

58. Statistical Power Factors that affect Power: 1-tail vs. 2-tailed β
Change from α = 0.05 two-tailed to α = 0.05 two-tailed
α = 0.05
p = 0.025 β
Power = 1 - β
p = 0.025
Reject H0
Fail to reject H0

59. Statistical Power Factors that affect Power: 1-tail vs. 2-tailed β
Change from α = 0.05 one-tailed to α = 0.05 two-tailed
Two tailed functionally cuts the α-level in half, which decreases the power.
α = 0.05
p = 0.025 β
Power = 1 - β
p = 0.025
Reject H0
Fail to reject H0

60. Statistical Power Factors that affect Power:
α-level: So as the a level gets smaller, so does the Power of the test
Sample size: As the sample gets bigger, the standard error gets smaller and the Power gets larger
Population standard deviation: As the population standard deviation gets smaller, the standard error gets smaller and the Power gets larger
Effect size: As the effect gets bigger, the Power gets larger
1-tail vs. 2-tailed: Two tailed functionally cuts the α-level in half, which decreases the power

61. Why care about Power? Determining your sample size
Using an estimate of effect size, and population standard deviation, you can determine how many participants need to achieve a particular level of power
When a result if not statistically significant
Is is because there is no effect, or not enough power
When a result is significant
“Statistical significance” versus “practical significance”

62. Ways of Increasing Power