How hypothesis testing works The Right Questions about Statistics: How hypothesis testing works Maths Learning Centre The University of Adelaide A hypothesis test is designed to DECIDE the answer to a YES OR NO question using DATA. You use probability to calculate how consistent your data is with a particular answer.

Is the median number of chapters in a novel = 20? Randomly choose 12 books

Book # Chapters 1 21 2 16 3 14 4 18 5 6 7 19 8 15 9 17 10 13 11 12 20 How likely is this to happen if the median for all books really is 20? How likely is this to happen if the median for all books really is 20? To answer this, I need to know what all the other possibilities are and compare... That’s too hard. So let’s turn all this data into just one number... c = 0.91 0.2 0.4 0.6 0.8 1.0 “TEST STATISTIC”

Suppose the median number of chapters for all books really is 20. And imagine what would happen if we took a different sample. “NULL HYPOTHESIS” 0.2 0.4 0.6 0.8 1.0

Book # Chapters 1 22 2 3 17 4 15 5 19 6 7 18 8 20 9 25 10 11 12 21 c = 0.55 0.2 0.4 0.6 0.8 1.0

Book # Chapters 1 23 2 17 3 26 4 21 5 25 6 7 24 8 9 12 10 20 11 15 c = 0.27 0.2 0.4 0.6 0.8 1.0

Book # Chapters 1 16 2 22 3 15 4 14 5 18 6 17 7 19 8 30 9 21 10 20 11 29 12 c = 0.55 0.2 0.4 0.6 0.8 1.0

Book # Chapters 1 16 2 14 3 29 4 15 5 25 6 7 8 17 9 13 10 11 23 12 20 c = 0.64 0.2 0.4 0.6 0.8 1.0

Book # Chapters 1 14 2 21 3 20 4 19 5 26 6 15 7 8 12 9 27 10 18 11 22 c = 0.55 0.2 0.4 0.6 0.8 1.0

Book # Chapters 1 15 2 3 21 4 20 5 12 6 7 22 8 18 9 10 11 13 c = 0.73 0.2 0.4 0.6 0.8 1.0

Book # Chapters 1 29 2 18 3 16 4 25 5 13 6 26 7 30 8 20 9 10 12 11 24 c = 0.36 0.2 0.4 0.6 0.8 1.0

Book # Chapters 1 20 2 18 3 12 4 21 5 17 6 15 7 8 9 30 10 13 11 23 c = 0.73 0.2 0.4 0.6 0.8 1.0

Book # Chapters 1 32 2 19 3 28 4 20 5 14 6 25 7 27 8 24 9 12 10 11 21 15 c = 0.45 0.2 0.4 0.6 0.8 1.0

Book # Chapters 1 25 2 17 3 4 23 5 6 16 7 8 18 9 10 20 11 13 12 15 c = 0.64 0.2 0.4 0.6 0.8 1.0

Book # Chapters 1 ## 2 3 4 5 6 7 8 9 10 11 12 c = 0.18 c = 0.82 c = 0.55 c = 0.45 c = 0.64 c = 0.27 c = 0.55 c = 0.45 0.2 0.4 0.6 0.8 1.0

“DISTRIBUTION OF TEST STATISTIC” This describes all the possibilities for what the test statistic could have been and how likely they all are. We can usually go straight to this distribution if we use some statistical theory 0.2 0.4 0.6 0.8 1.0 “DISTRIBUTION OF TEST STATISTIC”

I need to chose a cut-off for when I declare this to be unlikely. NOW: How likely is my test statistic if the median for all books really is 20? That is, what percentage of the possible test statistics are just as bad as mine? p = 0.01 “P VALUE” I need to chose a cut-off for when I declare this to be unlikely. Let’s choose 0.05. p = 0.01 is less than 0.05 So our test statistic unlikely. So conclude the median is not 20. 0.2 0.4 0.6 0.8 1.0

Let’s go over that again… Book # Chapters 1 21 2 16 3 14 4 18 5 6 7 19 8 15 9 17 10 13 11 12 20 Is the median number of chapters = 20? Take a random sample of books. Calculate a test statistic. c = 0.91 Figure out the distribution if the median really is 20. B(11,0.5) Calculate a P-value. p = 0.01 Decide the answer. The median for all books is not 20. “HYPOTHESIS TEST” 0.2 0.4 0.6 0.8 1.0 “SIGN TEST”

Another hypothesis test Book Weight (g) 1 242 2 366 3 424 4 312 5 307 6 238 7 317 8 265 9 10 314 11 217 12 379 Is the mean weight equal to 300 g? Take a random sample of books. Calculate a test statistic. t = 0.46 Figure out the distribution if the mean really is 300g. t (11) Calculate a P-value. p = 0.65 Decide the answer. The mean for all books could be 300g. “ONE-SAMPLE T-TEST” -2.0 -1.0 0.0 1.0 2.0

Another hypothesis test Book Title has “the” 1 No 2 Yes 3 4 5 6 7 8 9 10 11 12 Is the percentage of books with “the” in the title 50% ? Take a random sample of books. Calculate a test statistic. x = 3 Figure out the distribution if the percentage really is 50%. B(12,0.5) Calculate a P-value. p = 0.09 Decide the answer. The percentage with “the” could be 50%. “EXACT TEST FOR ONE PROPORTION” 2 4 6 8 10

Another hypothesis test Is the standard deviation of weights equal to 40g? Book Weight (g) 1 242 2 366 3 424 4 312 5 307 6 238 7 317 8 265 9 10 314 11 217 12 379 Take a random sample of books. Calculate a test statistic. χ2 = 22.9 Figure out the distribution if the standard deviation really is 40g. χ2(11) Calculate a P-value. p = 0.03 Decide the answer. The standard deviation of weights is not 40g. 5 10 15 20 25 “CHI-SQUARED TEST FOR ONE STANDARD DEVIATION”

Another hypothesis test Are SciFi books the same thickness as other books? Book Genre Thickness (cm) 1 SciFi 4.2 2 Other 2.4 3 3.6 4 2.7 5 3.9 6 2.9 7 3.7 8 3.0 9 4.0 10 2.0 11 12 2.8 Take a random sample of books. Calculate a test statistic. t = 6.62 Figure out the distribution if the average thickness really is the same. t(20) Calculate a P-value. p = 0.0001 Decide the answer. SciFi books are not the same thickness as other books. 2 6 4 -6 -2 -4 “TWO-SAMPLE T-TEST”

Another hypothesis test Are SciFi books more likely to be written by men? Book Genre Author Gender 1 SciFi Male 2 Other Female 3 4 5 6 7 8 9 10 11 12 Take a random sample of books. Calculate a test statistic. χ2 = 1.37 Figure out the distribution if the average thickness really is the same. χ2(1) Calculate a P-value. p = 0.24 Decide the answer. SciFi books are not more likely to be written by men. 6 8 12 10 4 2 “CHI-SQUARED TEST FOR ASSOCIATION”

So this is how you do a hypothesis test: Have a yes-or-no question. (One of the answers is the “null hypothesis”.) Collect data. Calculate a test statistic. Figure out what the distribution would be if you suppose the null hypothesis. Calculate a p-value. Decide the answer based on the p-value.

And this is how you interpret hypothesis test: A low p-value means your data is not very consistent with the null hypothesis and you don’t believe you’re in that situation. A high p-value means your data is consistent with the null hypothesis and you do believe you could possibly be in that situation. very strong evidence that the null is false strong evidence that the null is false some evidence that the null is false. no evidence that the null is false. 0.01 0.05 0.10 significance level reject null retain null