Statistical Tests - Power

Power The probability that a test rejects the null hypothesis when the null is false is called the power of the test against that alternative. Power = 1-β where β is the probability of a type 2 error.

Components There are four components that influence the conclusions you reach from a statistical test : Sample size, or the number of units in the study Effect size, or the salience of the treatment relative to the noise in measurement Significance level (alpha level), or the probability that the observed result is due to chance Power, or the probability that you will observe a treatment effect when it occurs Given values for any three of these components, it is possible to compute the value of the fourth.

Relationship between α, β, and Power α is the type 1 error. α = .01 means you have a 1 % chance of rejecting the null when it is true. Increasing α (e.g., from .01 to .05 or .10) increases the chances of making a type I error (i.e., saying there is a difference when there is not), and decreases the chances of making a type II error (i.e., saying there is no difference when there is) Increasing α (e.g., from .01 to .05 or .10) increases power because one will be rejecting the null more often and, consequently, when the alternative is true, there is a greater chance of believing it is true (i.e., power)

Example In a criminal trial, the defendant is considered not guilty until proven guilty beyond a reasonable doubt. H0: Defendant is not guilty HA: Defendant is guilty We should reject the null only if there is very strong evidence against it. Should we use a significance level of .01 or .20? Describe type 1 and 2 error and which is more serious. What is meant by power for the level you chose in #1?

Example - continued α =.01 is preferred , as the burden of proof would be heavier for finding the defendant guilty Type 1 error is more serious. This would mean convicting an innocent man. Type 2 error is the probability of deciding a guilty man is not guilty. Assume we choose a significance level of .01. the power of this test is the probability at α =.01 that the defendant will be found guilty when he is actually guilty.