What’s in the Bag? Lecture 4 Section Wed, Jan 25, 2006

How Strong is the Evidence? Rather than give an accept/reject answer, we may ask a different question: Rather than give an accept/reject answer, we may ask a different question: How strong is the evidence against H 0 ? How strong is the evidence against H 0 ?

The p-value p-value – The likelihood of getting by chance a value at least as extreme as the one observed if H 0 is true. p-value – The likelihood of getting by chance a value at least as extreme as the one observed if H 0 is true.

Two Bags If the selected token is worth $50, what is the p- value? If the selected token is worth $50, what is the p- value?

Two Bags Bag A Bag B

Two Bags Bag A Bag B At least as extreme as 50

Two Bags Bag A Bag B p-value = 2/20 = 0.10 At least as extreme as 50

A Two-Sided Test Bag F Bag E

A Two-Sided Test If the selected token is worth $8, what is the p- value? If the selected token is worth $8, what is the p- value? First, what is the direction of extreme? First, what is the direction of extreme? Which values are at least as extreme as 8? Which values are at least as extreme as 8? Bag E

A Two-Sided Test Bag E Bag F At least as extreme as 8

A Two-Sided Test Bag E Bag F p-value = 12/30 = 0.40

A Two-Sided Test In a two-sided test, if the null distribution is symmetric, then you can compute the probability in one direction, and then double it to get the p-value. In a two-sided test, if the null distribution is symmetric, then you can compute the probability in one direction, and then double it to get the p-value.

The p-value A small p-value is strong evidence against the null hypothesis. A small p-value is strong evidence against the null hypothesis. Why? Why? A large p-value is evidence in favor of the null hypothesis. A large p-value is evidence in favor of the null hypothesis.

The p-value Put differently, a small p-value is statistically significant. Put differently, a small p-value is statistically significant. A large p-value is not statistically significant. A large p-value is not statistically significant. This may seem counterintuitive since, generally speaking, small things are insignificant and large things are significant. This may seem counterintuitive since, generally speaking, small things are insignificant and large things are significant.

The p-value Prevalence and Cardiovascular Disease Correlates of Low Cardiorespiratory Fitness in Adolescents and Adults Prevalence and Cardiovascular Disease Correlates of Low Cardiorespiratory Fitness in Adolescents and Adults Prevalence and Cardiovascular Disease Correlates of Low Cardiorespiratory Fitness in Adolescents and Adults Prevalence and Cardiovascular Disease Correlates of Low Cardiorespiratory Fitness in Adolescents and Adults

Two Explanations of Unusual Observations The null hypothesis leads us to a certain expectation of what the data will show. The null hypothesis leads us to a certain expectation of what the data will show. Allowing for some randomness, if the data are close to our expectation, then we have no reason to doubt the null hypothesis. Allowing for some randomness, if the data are close to our expectation, then we have no reason to doubt the null hypothesis. But if the data deviate far from our expectation, then we doubt the truth of the null hypothesis. But if the data deviate far from our expectation, then we doubt the truth of the null hypothesis.

Two Explanations of Unusual Observations Under the null hypothesis, such a deviation would be due to chance. Under the null hypothesis, such a deviation would be due to chance. The p-value measures the likelihood of a deviation as large as the one we observed if the null hypothesis is true. The p-value measures the likelihood of a deviation as large as the one we observed if the null hypothesis is true. Small deviations are likely. Small deviations are likely. Large deviations are unlikely. Large deviations are unlikely.

Two Explanations of Unusual Observations There are two potential explanations for any deviation: There are two potential explanations for any deviation: H 0 is true and the deviation occurred by chance. H 0 is true and the deviation occurred by chance. H 0 is false. H 0 is false. Given the evidence, i.e., the size of the deviation, which explanation is more believable? Given the evidence, i.e., the size of the deviation, which explanation is more believable?

Example Consider the Intelligent Design hypothesis vs. the Evolution hypothesis. Consider the Intelligent Design hypothesis vs. the Evolution hypothesis. Which hypothesis uses the “chance” explanation? Which hypothesis uses the “chance” explanation? Which hypothesis uses a non-random, directed mechanism as the explanation? Which hypothesis uses a non-random, directed mechanism as the explanation? Which would be the null hypothesis? Which would be the null hypothesis? Which would have the burden of proof? Which would have the burden of proof?