The Practice of Statistics Third Edition Chapter 11: Testing a Claim Copyright © 2008 by W. H. Freeman & Company Daniel S. Yates

Section 11.4 Using Inference to Make Decisions

p(Type I error) = α p(Type II error) = β TRUTH TABLE (Power = 1 - β)

The blue area is the probability of a Type I error, α. The green area is the probability of a Type II error, β. Figure 11.9

P-value assumes H o is true. Power assumes some alternative H a is true.

Power High power is desirable The higher power the power, the more sensitive the test is and the better the test is at rejecting H o when H a is true. 80% power is becoming a standard.

Increasing the Power Increase α Increase sample size, n Consider a particular alternative that is farther away from μ o Decrease σ Best advice: pick the highest α you are willing to risk and get as large a sample as you can afford.

Section 11.3 Use and Abuse of Tests

Choosing a Significance Level The purpose of a test of significance is to give a clear statement of the degree of evidence provided by the sample against the null hypothesis. The p- value does this. Significance level is subjective. There is no sharp border between statistically significant and statistically insignificant, only increasingly strong evidence as the p-value decreases.

Statistical Significance vs. Practical Significance When a null hypothesis can be rejected, there is good evidence that an effect is present. But that effect maybe very small. When large samples are available, even tiny deviations from the null hypothesis will be significant. Statistical significance is not the same thing as practical significance (Ex 11.13) Look carefully at the actual data. The foolish user of statistics who feeds the data to a calculator or computer without exploratory analysis will often be embarrassed. Outliers can have great impact on significance. Include a CI for the parameter, it will estimate the size of an effect rather than simply asking if the test statistic is too large to reasonably occur by chance alone.

Don’t Ignore Lack of Significance Consider that there may be an effect from a treatment even if a finding isn’t statistically significant and, vice versa, there may not be an effect even if a finding is statistically significant. (Ex 11.14) Some small effects are only detectable with large sample sizes. When planning a study, verify that the test you plan to use has a high probability of detecting an effect of the size you hope to find.

Statistical Inference is Not Valid for All Sets of Data Badly designed survey or experiments often produce invalid results. Formal statistical inference cannot correct basic flaws in the design. (No garbage data) Hawthorne effect: workers change behavior when they know they are being studied. A randomized comparative experiment would isolate the actual effect from the treatment. Ex The sample should be randomly drawn from the population of interest. Randomization ensures that the rules of probability may be applied. Since inference relies on probability, we must have confidence in the use of probability to describe the data.

Beware of Multiple Analyses The reasoning behind statistically significance works well if you decide what effect you are seeking, design a study to search for it, and use a test of significance to weigh the evidence you get. Remember that α = 0.05 means you will get a significant finding 1 out of 20 times simply due to chance variation. Ex Many tests run at once will probably produce some significant results by chance alone.

Always Ask Details About Data Before Trusting Inference Not using all data can introduce bias. (Dropping out example)