1. Another Example
Consider a very popular computer game played by millions of people all over the world. The average score of the game is known to be 𝜇=5000 and the standard deviation is known to be σ=1000. Suppose the game developers have just created an updated version and want to know if it is more difficult than the original version. They plan to conduct a trial of the updated version on a random sample of 𝑛=100 players and perform a hypothesis test on the sample mean using 𝛼=0.05.
𝐻 0 : 𝜇=5000
𝐻 1 : 𝜇<5000
What is the probability of a Type I error (rejecting 𝐻 0 when, in fact, it is true)?
P(Type I Error) =𝛼=0.05
Suppose the updated version actually is more difficult, and the true mean is
𝜇 1 =4700. What is the probability of a Type II error (accepting 𝐻 0 when,
in fact, it is false)?
𝑥 𝛼 = −𝑧 𝛼 𝜎 𝑛 + 𝜇 0 → 𝑥 =− =4835.5
𝑧 𝛽 = 𝑥 𝛼 − 𝜇 1 𝜎/ 𝑛 = − =1.355
𝛽=𝑃 𝑍> 𝑧 𝛽 =𝑃 𝑍>1.355 =0.0877
𝐻 1
𝐻 0
𝜇 1 =4700
𝜎/ 𝑛 =100
𝜇 0 =5000
𝜎/ 𝑛 =100
𝜶=𝟎.𝟎𝟓
𝜷=𝟎.𝟎𝟖𝟕𝟕
𝑥 =4835.5

2. Another Example
Consider a very popular computer game played by millions of people all over the world. The average score of the game is known to be 𝜇=5000 and the standard deviation is known to be σ=1000. Suppose the game developers have just created an updated version and want to know if it is more difficult than the original version. They plan to conduct a trial of the updated version on a random sample of 𝑛=100 players and perform a hypothesis test on the sample mean using 𝛼=0.05.
𝐻 0 : 𝜇=5000
𝐻 1 : 𝜇<5000
What is the probability of a Type I error (rejecting 𝐻 0 when, in fact, it is true)?
P(Type I Error) =𝛼=0.05
Suppose the updated version actually is more difficult, and the true mean is
𝜇 1 =4700. What is the probability of a Type II error (accepting 𝐻 0 when,
in fact, it is false)?
𝑥 𝛼 = −𝑧 𝛼 𝜎 𝑛 + 𝜇 0 → 𝑥 =− =4835.5
𝑧 𝛽 = 𝑥 𝛼 − 𝜇 1 𝜎/ 𝑛 = − =1.355
𝛽=𝑃 𝑍> 𝑧 𝛽 =𝑃 𝑍>1.355 =0.0877
Power = 1−𝛽=1−0.0877=0.9123
𝐻 1
𝐻 0
Power =𝟎.𝟗𝟏𝟐𝟑
𝜇 1 =4700
𝜎/ 𝑛 =100
𝜇 0 =5000
𝜎/ 𝑛 =100
𝜷=𝟎.𝟎𝟖𝟕𝟕
𝑥 =4835.5

3. Same Example, Smaller 𝛼
Consider a very popular computer game played by millions of people all over the world. The average score of the game is known to be 𝜇=5000 and the standard deviation is known to be σ=1000. Suppose the game developers have just created an updated version and want to know if it is more difficult than the original version. They plan to conduct a trial of the updated version on a random sample of 𝑛=100 players and perform a hypothesis test on the sample mean using 𝛼=0.01.
𝐻 0 : 𝜇=5000
𝐻 1 : 𝜇<5000
What is the probability of a Type I error (rejecting 𝐻 0 when, in fact, it is true)?
P(Type I Error) =𝛼=0.01
Suppose the updated version actually is more difficult, and the true mean is
𝜇 1 =4700. What is the probability of a Type II error (accepting 𝐻 0 when,
in fact, it is false)?
𝑥 𝛼 = −𝑧 𝛼 𝜎 𝑛 + 𝜇 0 → 𝑥 =− =4767.4
𝑧 𝛽 = 𝑥 𝛼 − 𝜇 1 𝜎/ 𝑛 = − =0.674
𝛽=𝑃 𝑍> 𝑧 𝛽 =𝑃 𝑍>0.674 =0.2502
𝐻 1
𝐻 0
𝜇 1 =4700
𝜎/ 𝑛 =100
𝜇 0 =5000
𝜎/ 𝑛 =100
𝜷=𝟎.𝟐𝟓𝟎𝟐
𝜶=𝟎.𝟎𝟏
𝑥 =4767.4

4. Same Example, Smaller 𝛼
Consider a very popular computer game played by millions of people all over the world. The average score of the game is known to be 𝜇=5000 and the standard deviation is known to be σ=1000. Suppose the game developers have just created an updated version and want to know if it is more difficult than the original version. They plan to conduct a trial of the updated version on a random sample of 𝑛=100 players and perform a hypothesis test on the sample mean using 𝛼=0.01.
𝐻 0 : 𝜇=5000
𝐻 1 : 𝜇<5000
What is the probability of a Type I error (rejecting 𝐻 0 when, in fact, it is true)?
P(Type I Error) =𝛼=0.01
Suppose the updated version actually is more difficult, and the true mean is
𝜇 1 =4700. What is the probability of a Type II error (accepting 𝐻 0 when,
in fact, it is false)?
𝑥 𝛼 = −𝑧 𝛼 𝜎 𝑛 + 𝜇 0 → 𝑥 =− =4767.4
𝑧 𝛽 = 𝑥 𝛼 − 𝜇 1 𝜎/ 𝑛 = − =0.674
𝛽=𝑃 𝑍> 𝑧 𝛽 =𝑃 𝑍>0.674 =0.2502
Power = 1−𝛽=1−0.2502=0.7498
𝐻 1
𝐻 0
Power =𝟎.𝟕𝟒𝟗𝟖
𝜇 1 =4700
𝜎/ 𝑛 =100
𝜇 0 =5000
𝜎/ 𝑛 =100
𝜷=𝟎.𝟐𝟓𝟎𝟐
𝑥 =4767.4

5. Same Example, Larger n
Consider a very popular computer game played by millions of people all over the world. The average score of the game is known to be 𝜇=5000 and the standard deviation is known to be σ=1000. Suppose the game developers have just created an updated version and want to know if it is more difficult than the original version. They plan to conduct a trial of the updated version on a random sample of 𝑛=200 players and perform a hypothesis test on the sample mean using 𝛼=0.01.
𝐻 0 : 𝜇=5000
𝐻 1 : 𝜇<5000
What is the probability of a Type I error (rejecting 𝐻 0 when, in fact, it is true)?
P(Type I Error) =𝛼=0.01
Suppose the updated version actually is more difficult, and the true mean is
𝜇 1 =4700. What is the probability of a Type II error (accepting 𝐻 0 when,
in fact, it is false)?
𝑥 𝛼 = −𝑧 𝛼 𝜎 𝑛 + 𝜇 0 → 𝑥 =− =4835.5
𝑧 𝛽 = 𝑥 𝛼 − 𝜇 1 𝜎/ 𝑛 = − / =1.916
𝛽=𝑃 𝑍> 𝑧 𝛽 =𝑃 𝑍>1.916 =0.0277
𝐻 1
𝐻 0
𝜇 1 =4700
𝜎/ 𝑛 =70.71
𝜇 0 =5000
𝜎/ 𝑛 =70.71
𝜶=𝟎.𝟎𝟏
𝜷=𝟎.𝟎𝟐𝟕𝟕
𝑥 =4835.5

6. Same Example, Larger n
Consider a very popular computer game played by millions of people all over the world. The average score of the game is known to be 𝜇=5000 and the standard deviation is known to be σ=1000. Suppose the game developers have just created an updated version and want to know if it is more difficult than the original version. They plan to conduct a trial of the updated version on a random sample of 𝑛=200 players and perform a hypothesis test on the sample mean using 𝛼=0.01.
𝐻 0 : 𝜇=5000
𝐻 1 : 𝜇<5000
What is the probability of a Type I error (rejecting 𝐻 0 when, in fact, it is true)?
P(Type I Error) =𝛼=0.01
Suppose the updated version actually is more difficult, and the true mean is
𝜇 1 =4700. What is the probability of a Type II error (accepting 𝐻 0 when,
in fact, it is false)?
𝑥 𝛼 = −𝑧 𝛼 𝜎 𝑛 + 𝜇 0 → 𝑥 =− =4835.5
𝑧 𝛽 = 𝑥 𝛼 − 𝜇 1 𝜎/ 𝑛 = − / =1.916
𝛽=𝑃 𝑍> 𝑧 𝛽 =𝑃 𝑍>1.916 =0.0277
Power = 1−𝛽=1−0.2502=0.9723
𝐻 1
𝐻 0
Power =𝟎.𝟗𝟕𝟐𝟑
𝜇 1 =4700
𝜎/ 𝑛 =70.71
𝜇 0 =5000
𝜎/ 𝑛 =70.71
𝜷=𝟎.𝟎𝟐𝟕𝟕
𝑥 =4835.5

7. Summary

8. Summary

9. Summary

10. Practical Implications
Type I Error, Type II Error, and Power are crucial considerations in any hypothesis testing study
For a more consequential example, consider a new medical treatment developed to cure a life threatening disease. The treatment is tested on a random sample of ill patients.
Type I Error: Deciding that the treatment works when, in fact, it doesn’t. This would be a dangerous mistake to make, because patients would be given this ineffective treatment.
Type II Error: Deciding that the treatment does not work when, in fact, it does. This would be a tragic mistake, since this life- saving treatment would not be given to patients.
Power: Probability of deciding that the treatment works when, in fact, it does. This would be the correct decision that we want to make.
Ideally, we want to minimize both Type I error and Type II error (and maximize power). However, as explained earlier, the probabilities of Type I error and Type II error are inversely related: as one goes down, the other goes up. Therefore, a compromise must be made in choosing a low enough probability of Type I error (𝛼) while keeping the probability of Type II error (𝛽) in check.
For any given study, the sample size should be large enough to have adequate power (or low 𝛽). If the sample size is too small, we would most likely not be able to detect a significant effect even it does exist in reality.