Lesson Use and Abuse of Tests

Knowledge Objectives Distinguish between statistical significance and practical importance Identify the advantages and disadvantages of using P-values rather than a fixed level of significance

Construction Objectives none this lesson

Vocabulary Statistically Significant – when observed results are unlikely under the assumption that the null hypothesis is true. When results are found to be statistically significant, we reject the null hypothesis Practical Significance – refers to things that are statistically significant, but the actual difference is not large enough to cause concern or be considered important

Choosing a Level of Significance How small a P-value is convincing evidence against the null hypothesis? This depends on two circumstances: How plausible is H 0 ? -- if H 0 represents something people have believed for years, then you will need strong evidence (a small P- value) What are the consequences of rejecting H 0 ? -- if rejecting H 0 costs the company lots of money, then you will need strong evidence. There is no sharp border between “statistically significant” and “statistically insignificant”

Example 1 Your company has developed a new antibacterial cream. From previous research you know that with no medication, a small cut on the inner forearm will heal on average in 7.6 days with a standard deviation of 1.4 days. You want to test this new cream at the 5% significance level. We cut 25 volunteer college students and apply the cream to the wounds. The mean healing time for them was 7.1 days. We will assume that σ = 1.4 days.

Example 1 cont Hypothesis: H 0 : H a : Conditions: 1: 2: 3: μ = 7.6 days (to heal the cut) μ < 7.6 days SRS -- assume that our volunteers represent a simple random sample (shaky for general population vs college students) Normality is not considered a large enough number for CLT to apply. We would need to check graphs (box-plot and normality plot) to verify reasonableness of normality Independence -- easy to assume each test is independent for the others

Example 1 cont Calculations: Interpretation: x – μ – 7.6 Z 0 = = σ/√n 1.4/√25 = P-value = P(z < Z 0 ) = P(z < -1.79) = (unusual !) P-value =  so only 4% of the time could we get a more extreme value. Since this is less than α = 0.05, we reject H 0 and conclude that the cream promotes quicker healing While statistically significant, a ½ day sooner healing is not practically important.

Significance Statistical Significance is not the same thing as practical importance. Large sample sizes can promote statistical significance with not practical importance. Remember to graphically examine your data. Outliers can make you conclude the wrong results if you blindly follow the common significance tests. Examine confidence intervals, since they provide estimates of the effect without judging whether it is too large to occur by chance.

Other Warnings Over Analysis –Remember, even with a 5% level of significance, one in 20 studies is likely to show the unusual Remember our test’s required conditions –Tests may give odd results if conditions not met –Garbage In = Garbage Out (GIGO) Remember the population of interest and don’t generalize beyond them

Significance Tests Remember our 3 conditions: SRS – simple random samples –not all of our sampling techniques fit this Normality –samples too small for Central Limit Theorem to apply –samples with outliers or extremely skewed distributions Independence –hard to get large samples from small populations Hawthorne effect – knowledge of a study increases short- term productivity (watching effects the watched)

Summary and Homework Summary –P-values are more informative than a simple reject or fail to reject conclusion –Very small effects can be highly significant, especially with a very large sample size –Statistical significance does not mean practical importance –Significance tests are not always valid; GIGO! Homework –pg