1. Chapter 12
Power Analysis

2. Power Analysis
A statistical technique that measures the sensitivity of an experiment.
In correlation, sensitivity is the ability to detect a relationship between two variables above and beyond sampling fluctuation.
In an experiment, sensitivity is the ability to detect effects of the independent variable, differences among the group means that cannot be accounted for by sampling fluctuation.

3. Power Analysis
Used to determine how many subjects are needed to obtain the desired sensitivity in the experiment.
Used to decide if we should believe that non-significant findings really indicate that the null hypothesis is true.

4. Statistics and Significance
When we use statistics to test the null hypothesis ...
We are trying to avoid making a prediction, based on a relationship in a sample, that does not actually hold up in the whole population.
But mistakes (Type 1 errors) are possible with statistical tests.

5. Type 1 errors
We will mistakenly obtain misleading significant results in 5 per cent of the studies we analyze.
This happens when sampling fluctuation alone produces results outside the range predicted by the null.
This can happen whether we are using r, F, or t (or any other test statistic) to test the null.

6. Type 1 error and alpha
Type 1 Error - Rejecting the null hypothesis when the null hypothesis is true.
Alpha is the probability of making a Type 1 Error.
We set alpha directly and thus determine the critical value of test statistics such as r, F, t, Dunnett’s t and HSD.
By convention, in the social and bio-medical sciences, alpha= .05 (almost always).

7. Type 2 error and beta
Type 2 Error – Failing to reject the null hypothesis when the null hypothesis is false.
Therefore not generalizing from the pattern of results seen in your study to the population as a whole.
Beta is the probability of making a Type 2 Error.
Beta is determined by:
The number of research participants in the study
The design of the study
The size of the effect of an independent variable or correlation in the population as a whole.
2/23/2019
Ch12

8. The Experimental Results
Hypothesis Testing
The Experimental Results
Reject H0.
Fail to reject H0.
The Real
Situation
in the
Population
H0 is true.
H0 is false.
Type 1 Error OK OK
Type 2 Error

9. Tradeoffs between Type 1 and Type 2 errors
If we are too conservative (alpha = .001), we will rarely find an effect mistakenly, but we will also get significant findings very seldom. So we will get a lot of nonsignificant findings and we will often miss variables that would make a real differences in the population or correlations that occur in the population as well as in the sample.
If we are too liberal, (alpha = .20), we will tend to generalize to the population from differences in samples that occur simply because of sampling fluctuation. We will get lots of significant findings, but many will be mistaken.

10. A compromise between Type 1 and Type 2 errors
Setting alpha at .05 allows a reasonable compromise for the two types of errors.
5 studies in 100 will yield significant findings even though the null is true and only sampling fluctation is at work.
This is stringent enough to be cautious, but not so stringent that you can practically never find significance and therefore make lots of Type 2 errors.

11. Determining Beta Beta = the odds on making a Type 2 error
But, how do we determine beta?
Remember, you can only make a Type 2 error if
You don’t get significant results
AND
The null is false and there really is an effect in the population similar to the one in your sample.

12. Many factors effect beta
The value of alpha (already discussed):
Making alpha more stringent ( for example: alpha = .01) makes it less likely that we will make a Type 1 error. but more likely that we will make a Type 2 Error (increases beta).
Conversely, increasing alpha (e.g., alpha = .10) makes it more likely that we will make a Type 1 error. but less likely that we will make a Type 2 Error (decreases beta).
2/23/2019
Ch12

13. Factors affecting Beta
Random measurement problems and individual differences
Sloppy research increases noise due to measurement problems and therefore the size of MSW. This makes it harder to find a significant F ratio and so increases beta.
More heterogeneous groups make for greater individual differences and increase beta.
Conversely, the “repeated measures” design uses the same subjects in each group. This gets rid of individual differences, makes MSW smaller, increases the chance of getting significant findings, and so decreases beta.

14. … Factors affecting Beta
The strength of the relationship or effect:
It is easier to detect strong relationships and big effects. As a result, we are less likely to make a Type 2 Error when big, strong effects are present.
It is more difficult to detect weak relationships, therefore we are more likely to make a Type 2 Error when trying to document one.

15. Study big, robust effects.
Research tip
Study big, robust effects.
2/23/2019
Ch12

16. … Factors affecting Beta
The number of subjects:
The more subjects in a sample, the more likely it is that a true effect will be seen because
A. the critical values of test statistics get less extreme and
B. because as df increase, the sample will resemble the population more and more.

17. Power = 1 - Beta
Beta is the probability that we will get nonsignificant results although we would find a correlation or an effect if we could measure the entire population
When that happens we will incorrectly fail to reject the null hypothesis.
The reverse side of beta is called power.
Power is the probability that a study will correctly reject a null hypothesis.
Power = 1 - beta
If beta goes up; power must go down.
If power goes up; beta must go down.
A scientist wants to run studies with lots of power.

18. Power
Imagine we have an experiment where we do not find any significant results.
If an experiment has enough power, it is reasonable to conclude that there is, in fact, no effect, or at least no moderate or large effect, of the independent variable.

19. Increasing Power
We can get more power by increasing the number of research participants in a study
Power tables can tell us:
What are a reasonable number of subjects to have a “powerful” experiment?
Does someone else’s experiment have enough subjects to believe their result of “no significant finding” means that there are is no effect of the IV or real correlation in the population?

20. Using the Power Tables
First, estimate the strength of the experimental effect. This is usually based on a pilot study or knowledge of the research literature.
Second, decide on the desired amount of power.
Third, read the number of subjects from the table.

21. Types of Power Tables
There are Power Tables for the various types of experiments.
We will look at:
Correlation
One-way, Independent Groups
Factorial Designs with Independent Groups

22. Power & Correlation Power .50 .70 .80 .90 .95
Inadequate Minimal Adequate Good Ideal
Power
rho Beta

23. Power & Correlation Power .50 .70 .80 .90 .95
Amount of Power
and beta across
the top.
Inadequate Minimal Adequate Good Ideal
Power
rho Beta
Strength of
correlation in
the population

24. Power & Correlation Power .50 .70 .80 .90 .95
Inadequate Minimal Adequate Good Ideal
Power
rho Beta
Number of
subjects.

25. Some Principles For correlation:
If possible, only do correlational studies with variables you believe to have a moderate or strong relationship (where rho is at least .25).
Only only do correlational studies with a large enough sample size to give you power of at least .70 and preferably .80 or better.

26. EXAMPLE A pilot study shows a correlation of .50.
How many subjects do you need to have a 20% chance of committing a Type 2 Error?

27. Power & Correlation Power .50 .70 .80 .90 .95
Inadequate Minimal Adequate Good Ideal
Power
rho Beta 28
You should have at
least 28 subjects.

28. Evaluating research
Someone did a study with 150 subjects and did not get a significant correlation. Can we believe that there is no correlation in the population?

29. Power & Correlation Power .50 .70 .80 .90 .95
Inadequate Minimal Adequate Good Ideal
Power
rho Beta
150 subjects gives this research a lot of power.
It is easy for us to believe that if no correlation was
found, then there is no strong or moderate size correlation
in the population. A weak correlation is still possible.

30. One Way Experiment with Independent Groups

31. 2 Amount of Power and beta across the top. 3 # of groups. # of groups
Inadequate Minimal Adequate Good Ideal
One-way
independent
groups
Power
Beta
NG Small
NG Medium
NG Large
# of groups. 2
NG Small
NG Medium
NG Large 3
# of
groups
NG Small
NG Medium
NG Large 4
NG Small
NG Medium
NG Large 5
NG Small
NG Medium
NG Large 6

32. 2 3 Size of effect. # of groups 4 5 6 One-way independent groups
Inadequate Minimal Adequate Good Ideal
One-way
independent
groups
Power
Beta
NG Small
NG Medium
NG Large 2
NG Small
NG Medium
NG Large 3
Size of effect.
# of
groups
NG Small
NG Medium
NG Large 4
NG Small
NG Medium
NG Large 5
NG Small
NG Medium
NG Large 6

33. of subjects in each group.
Inadequate Minimal Adequate Good Ideal
One-way
independent
groups
Power
Beta
NG Small
NG Medium
NG Large 2
Therefore, n is k * nG.
For example,
5 * 240 = 1200 subjects.
NG Small
NG Medium
NG Large 3
# of
groups
NG Small
NG Medium
NG Large 4
NG Small
NG Medium
NG Large 5
nG is the number
of subjects in each group.
NG Small
NG Medium
NG Large 6

34. Another Example Three groups of 27 subjects each.
No significant effect found.
What are the odds that the researcher would have found a large effect if it were there?
Is this size study large enough to be an adequate test for large effects?

35. This is a better than 2 adequate size to test for large effects.
Inadequate Minimal Adequate Good Ideal
One-way
independent
groups
Power
Beta
nG Small
nG Medium
nG Large 2
This is a better than
adequate size to test
for large effects.
3 groups.
9 out of 10
odds of finding
a large effect.
nG Small
nG Medium
nG Large 3
# of
groups
nG Small
nG Medium
nG Large 4
27 subjects
in each group.
nG Small
nG Medium
nG Large 5
nG Small
nG Medium
nG Large 6

36. Answer Good chance of finding a large effect, if it were there.
Probably no large effect present in the population.
Study is big enough to detect large effects.

37. Three groups of 27 subjects.
No significant effect found.
What are the odds that the researcher would have found a medium effect if it were there?
Is this size study large enough to be an adequate test for medium effects?

38. inadequate size to test
Inadequate Minimal Adequate Good Ideal
One-way
independent
groups
Power
Beta
NG Small
NG Medium
NG Large
This is an
inadequate size to test
for medium effects. 2
5 out of 10
odds of finding
a medium effect.
3 groups.
NG Small
NG Medium
NG Large 3
# of
groups
NG Small
NG Medium
NG Large 4
27 subjects
in each group.
NG Small
NG Medium
NG Large 5
NG Small
NG Medium
NG Large 6

39. Factorial Experiments with Independent Groups
Tables for factorial experiments can get very complicated.
I will only show a stripped down table.

40. moderate-size effects. There are no values for small or large effects.
Inadequate Minimal Adequate Good Ideal
Moderate-size effects,
factorial design,
independent groups
Power
Beta
2X2 NG
This table only shows
moderate-size effects.
There are no values for
small or large effects.
2X3 NG
# of
cells
2X4 NG
If there are an adequate amount of subjects for a medium effect, then there will also be an adequate amount for a large effect.
Small effects are usually too difficult to study.
3X4 NG
2X2X2 NG
2X2X3 NG

41. 2X2 2X3 This shows the # of Factorial design. 2X4 cells 3X4 2X2X2
Inadequate Minimal Adequate Good Ideal
Moderate-size effects,
factorial design,
independent groups
Power
Beta
2X2 NG
2X3 NG
This shows the
Factorial design.
# of
cells
2X4 NG
3X4 NG
2X2X2 NG
2X2X3 NG

42. the table shows the number of subjects in each group.
Inadequate Minimal Adequate Good Ideal
Moderate-size effects,
factorial design,
independent groups
Power
Beta
2X2
Since we are looking at
independent groups,
the table shows the number
of subjects in each group. nG
2X3 nG
# of
cells
2X4 nG
3X4 nG
n = # cells * nG.
n = 30*2*4 = 240 subjects.
2X2X2 nG
2X2X3 nG

43. A 2X2 Factorial design with independent groups.
How many subjects do you need to pick up an interaction with a medium effect size and have a beta = .05?

44. 2X2 2X3 # of 2X4 n = 54*2*2 = 216 subjects. cells 3X4 2X2X2 2X2X3
Inadequate Minimal Adequate Good Ideal
Moderate-size effects,
factorial design,
independent groups
Power
Beta
2X2 NG
2X3 NG
n = 54*2*2 = 216 subjects.
# of
cells
2X4 NG
3X4 NG
2X2X2 NG
2X2X3 NG

45. A 2X4 Factorial design with independent groups.
40 subjects in each cell for a total of 320 subjects.
No significant results were reported. What can we conclude?

46. 40 subjects is more than 36 subjects.
Inadequate Minimal Adequate Good Ideal
Moderate-size effects,
factorial design,
independent groups
Power
Beta
Conclusion: There is an ideal amount of power in this experiment and it is likely that if a medium or large effect were present that it would be detected.
2X2 NG
2X3 NG
# of
cells
2X4 NG
3X4 NG
40 subjects is more than 36 subjects.
2X2X2 NG
2X2X3 NG

47. Conclusion about nonsignificant results
Is the study large enough so that if there were a real effect in the population, we should have seen it?
If the study is big enough and we have sufficient power, but we get nonsignificant results, neither of the independent variables nor their interaction is likely to cause medium or large differences in the population as a whole.
2/23/2019
Ch12

48. Summary
Type 1 Error - Rejecting the null hypothesis when the null hypothesis is true.
Alpha is the probability of making a Type 1 Error.
Type 2 Error - Accepting the null hypothesis when the null hypothesis is false.
Beta is the probability of making a Type 2 Error.

49. … Conclusion Factors affecting beta: The value of alpha.
The noise in the experiment.
The strength of the effect.
The number of subjects.

50. … Conclusion Power is related to beta. Power Tables
As power goes up; beta must go down.
As power goes down; beta must go up.
Power Tables
Power tables are available for various experimental designs.
They tell us how many subjects to use.
They tell whether to believe a non-significant finding.

51. 2/23/2019
Ch12

52. 2/23/2019
Ch12