The Fine Art of Knowing How Wrong You Might Be

To Err Is Human Humans make an infinitude of mistakes. I figure safety in numbers makes it a little more okay. When we do a hypothesis test, there are only so many options for what we decide to do. More specifically, there are only four possible outcomes.

Four?!? Some of you might be thinking that our only two possible options are to reject H 0 or to not reject H 0. You are correct. In each of these cases, though, we might be right or we might be wrong. That is where the number four comes from.

Four We might find sufficient evidence, and it turns out the evidence is telling us the truth. We might find sufficient evidence, but it turns out the evidence was naughty, bad evidence that was misleading us. (A frame job.) We might not find sufficient evidence, but it turns out that there really is the difference we were looking for. We might not find sufficient evidence, and it turns out that is correct and that nothing is going on.

So Then What? No matter what decision we make, we might be wrong. When you’re right, you’re right. When you’re wrong, you don’t know it, so act like you’re right…it projects an attractive level of confidence. In statistics we have no real way to find the truth without taking a census. Do not administer a census if you can help it. A census is dumb.

Where Am I Going With This? If we reject H 0 (which means there was sufficient evidence), but we were wrong, this is called a Type I Error (Roman Numeral 1). The probability of making this kind of error is alpha. So, normally, 5%. We call this percentage significance level, and use the letter α for it. We can set it as high or low as we want. Normally we go with 5%.

Why Not Make It 0%? If we do not reject H 0 (which means there was not sufficient evidence to draw a conclusion), but something really is going on, this is called a Type II Error (Roman Numeral 2). The probability of this kind of error is called β and it is a real pain to calculate, so we are not going to. What you need to know about it is that the lower α is, the higher β is.

5% Is A Good Balance There are really only 2 things we can do to make β smaller. First, we can make α higher. This is usually counterproductive, as it just makes us more likely to be wrong the other direction. Second, we can make n (the sample size) larger. Larger sample sizes make everything better. Make the sample size too large, though, and you violate the less than 10% condition. Also, if the sample is too large, you are pretty much doing a census.

A Pair of Complements For a 2-sided test (One where the alternative is a ≠ kind), the complement of α is the confidence level. And the compliment of the confidence level is the significance level. So, 95% confidence matches with 5% (for a ≠ test). The complement of β is called statistical power. You don’t need the formula for this, and only need to know the term in a general way.

Significance Remember how we discussed the difference between statistical significance and practical significance? I remember it. Hopefully we can all remember it. If the p-value is lower than α, we found sufficient evidence. Another way to say this is that a “statistically significant difference was found”. Statistically significant means the H 0 got rejected.

Nothing To Prove Remember how I said earlier that Statistics is never used to prove anything ever? I remember it. Hopefully we can all remember it. We never prove the null or the alternative hypothesis. We only support the alternative strongly enough or do not support it strongly enough.

A Truckload of Apples On a recent homework problem you were asked to find out how likely it was that a sample of 150 apples were taken from a whole truckload would have less than 5% damaged. We were able to do this math because we were told that the true percentage of damaged apples was 8%.

A Truckload of Apples In the real world we would not know that 8% of the whole truckload was damaged. They just told us that so we could do the math. So what should we do in the real world? Since we don’t know what the true percent is, we will do the next best thing…we will make a wild assumption about the truth.

Wild Assumption In practice we will make a sensible assumption, actually. Our assumption will usually be based on something that is known. This could be past data. This could be the value from a similar group. This could be.5 in the case of fair coins. The purpose in making this assumption is, primarily, to allow us to do the math.

Alternatives The alternative hypothesis will be based on whatever we are trying to prove. We use the null hypothesis so we can do the math, but really the alternative hypothesis is what we are trying to focus on when we run a hypothesis test.

Foreshadowing FTW Tomorrow we will discuss inferior ways to run a hypothesis test. I say inferior, but they are perfectly acceptable. Mr. J.P. Damron, in fact, almost always favored a confidence interval method for running hypothesis tests. It is worth noting that he never once got a hypothesis test wrong with this method, and seriously did over 30 of them.

Assignments You can read chapter 21, but I do not think it is necessary, especially if you have limited free time. Test 4 is next week on Thursday. There will be another Quiz Retake Bonanza on February 19 th (the day after the test).