AP Statistics Section 11.3

Statistical significance is valued because it points to an effect that is __________ to occur simply by chance. Carrying out a significance test is often quite simple. Using tests wisely is not so simple. Keep these points in mind when using or interpreting significance tests.

There is no sharp border between “statistically significant” and “statistically insignificant,” only increasingly strong evidence as the P­- value ____________. Giving the p- value allows each individual to decide for themselves if the evidence is convincing.

Statistical Significance and Practical Importance When a null hypothesis, which means “no effect” or “no difference”, can be rejected at the usual levels ( ______ or ______ ), there is good evidence that an effect or a difference is, in fact, present. But that effect or difference may be very small. When large samples are available, even ________ deviations from the null hypothesis will be significant.

Statistical ____________is not the same thing as practical ____________.

The remedy for attaching too much importance to statistical significance is to pay attention to the ___________ as well as to the _________. Plot your data and examine them carefully. Are there ________ or other deviations from a common pattern? A few outlying observations can produce highly significant results if you blindly apply common significance tests. Outliers can also destroy the significance of otherwise- convincing data.

Don’t Ignore Lack of Significance There is a tendency to infer that there is no effect whenever a _______ fails to attain the usual ___ standard.

In some areas of research, _______ effects that are detectable only with _______ sample sizes can be of great practical significance. Data accumulated from a large number of patients taking a new drug may be needed before we can conclude that there are life-threatening consequences for a small number of people.

When planning a study, verify that the test you plan to use has a ______ probability of detecting an effect of the size you hope to find.

Statistical Inference Is Not Valid for All Sets of Data Formal statistical inference cannot correct flaws in the design of surveys or experiments. Each test is valid only in certain circumstances, with properly produced data being particularly important.

Example 1: You wonder whether background music would improve the productivity of a group of workers at your place of employment. You discuss the idea with the workers and then add music to the background. You find a statistically significant increase.

Don’t be impressed. In fact, almost any change in the work environment together with the knowledge that a study is under way will produce a short-term productivity increase. This is known as the __________________. The significance test informs you that an increase has occurred that is larger than would often arise by chance alone. It does not tell you what else, other than chance, caused the increase. The most plausible explanation is that workers change their behavior when they know they are being studied. Your experiment was uncontrolled so the significant result cannot be interpreted. A _______________________ experiment would isolate the actual effect of background music and so make significance meaningful.

Always ask how the data were produced. Don’t be too impressed by p-values on a printout until you are confident that the data deserve a formal analysis.