1. 6.1 The Role of Probability in Statistics: Statistical Significance
LEARNING GOAL
Understand the concept of statistical significance and the essential role that probability plays in defining it.
Page 234

2. Definition
A set of measurements or observations in a statistical study is said to be statistically significant if it is unlikely to have occurred by chance.
Page 234

3. EXAMPLE 1 Likely Stories?
A detective in Detroit finds that 25 of the 62 guns used in crimes during the past week were sold by the same gun shop. This finding is statistically significant.
Because there are many gun shops in the Detroit area, having 25 out of 62 guns come from the same shop seems unlikely to have occurred by chance.
Page 234

4. EXAMPLE 1 Likely Stories?
b. In terms of the global average temperature, five of the years between 1990 and 1999 were the five hottest years in the 20th century. Having the five hottest years in 1990–1999 is statistically significant.
By chance alone, any particular year in a century would have a 5 in 100, or 1 in 20, chance of being one of the five hottest years. Having five of those years come in the same decade is very unlikely to have occurred by chance alone.
This statistical significance suggests that the world may be warming up.
Page 234

5. EXAMPLE 1 Likely Stories?
c. The team with the worst win-loss record in basketball wins one game against the defending league champions.
This one win is not statistically significant because although we expect a team with a poor win-loss record to lose most of its games, we also expect it to win occasionally, even against the defending league champions.
Page 234

6. From Sample to Population
EXAMPLE 2 Statistical Significance in Experiments
A researcher conducts a double-blind experiment that tests whether a new herbal formula is effective in preventing colds. During a three-month period, the 100 randomly selected people in a treatment group take the herbal formula while the 100 randomly selected people in a control group take a placebo.
The results show that 30 people in the treatment group get colds, compared to 32 people in the control group. Can we conclude that the herbal formula is effective in preventing colds?
Pages

7. EXAMPLE 2 Statistical Significance in Experiments
Whether a person gets a cold during any three-month period depends on many unpredictable factors. Therefore, we should not expect the number of people with colds in any two groups of 100 people to be exactly the same.
In this case, the difference between 30 people getting colds in the treatment group and 32 people getting colds in the control group seems small enough to be explainable by chance.
So the difference is not statistically significant, and we should not conclude that the treatment is effective.
Solution:
Page 235

8. Quantifying Statistical Significance
The definition of statistical significance that we’ve been using so far is too vague. We need a way to quantify the idea of statistical significance.
In general, we determine statistical significance by using probability to quantify the likelihood that a result may have occurred by chance. We therefore ask a question like this one:
Is the probability that the observed difference occurred by chance less than or equal to 0.05 (or 1 in 20)?
Page 235

9. Quantifying Statistical Significance
If the answer is yes (the probability is less than or equal to 0.05), then we say that the difference is statistically significant at the 0.05 level.
If the answer is no, the observed difference is reasonably likely to have occurred by chance, so we say that it is not statistically significant.
The choice of 0.05 is somewhat arbitrary, but it’s a figure that statisticians frequently use. Nevertheless, other probabilities are sometimes used, such as 0.1 or 0.01.
Pages

10. Quantifying Statistical Significance
• If the probability of an observed difference occurring by chance is 0.05 (or 1 in 20) or less, the difference is statistically significant at the 0.05 level.
• If the probability of an observed difference occurring by chance is 0.01 (or 1 in 100) or less, the difference is statistically significant at the 0.01 level.
Page 235

11. TIME OUT TO THINK
Suppose an experiment finds that people taking a new herbal remedy get fewer colds than people taking a placebo, and the results are statistically significant at the 0.01 level. Has the experiment proven that the herbal remedy works? Explain.
Page 236

12. EXAMPLE 2 Polio Vaccine Significance
In the test of the Salk polio vaccine (see Section 1.1), 33 of the 200,000 children in the treatment group got paralytic polio, while 115 of the 200,000 in the control group got paralytic polio. Calculations show that the probability of this difference between the groups occurring by chance is less than Describe the implications of this result.
Solution: The results of the polio vaccine test are statistically significant at the 0.01 level, meaning that there is a 0.01 chance (or less) that the difference between the control and treatment groups occurred by chance.
Page 236

13. EXAMPLE 2 Polio Vaccine Significance
Solution: (cont.)
Therefore, we can be fairly confident that the vaccine really was responsible for the fewer cases of polio in the treatment group.
(In fact, the probability of the Salk results occurring by chance is much less than 0.01, so researchers were quite convinced that the vaccine worked; as we’ll discuss in Chapter 9, this probability is called a “P-value.”)
Page 236

14. The End