1. BIOL 582 Lecture Set 2 Inferential Statistics, Hypotheses, and Resampling

2. Statistics we have already learned (e.g., mean, variance, standard deviation) are descriptive statistics. Statistics that are tests for specific patterns in data using independent variables to generate hypotheses (Y vs. X) are inferential statistics (e.g., t-test, ANOVA, likelihood ratio tests). Procedure: Obtain an observed condition Compare observed condition to distribution of expected values (or their likelihood) Determine whether observed value is something interesting (i.e., it is unexpected relative to chance?) BIOL 582 Inferential Statistics

3. Inferential statistics compare observed value to distribution of ‘expected’ values Significance determined based upon how ‘extreme’ the observed value is relative to the distribution Resampling methods are one approach for generating expected distributions Observed value probability BIOL 582 Inferential Statistics Side Trip to learn about probability…

4. BIOL 582 Variant of the Fisher Exact Test Sir Ronald Aylmer Fisher (1890 –1962) Fisher’s exact test (named after Fisher for work he originally published in 1922, but also published in other ways after Is this meaningful?

5. BIOL 582 Variant of the Fisher Exact Test Sir Ronald Aylmer Fisher (1890 –1962) Fisher’s exact test (named after Fisher for work he originally published in 1922, but also published in other ways after Consider all possible ways to get 2 groups of 4 with these data m permutations; n values; g number of groups 28 possible ways to get 2 groups of 4!

6. BIOL 582 Variant of the Fisher Exact Test Sir Ronald Aylmer Fisher (1890 –1962) Fisher’s exact test (named after Fisher for work he originally published in 1922, but also published in other ways after Observed Random Do this for all 28 possible permutations and create a distribution of differences

7. 2 ways to get 2.5 BIOL 582 Variant of the Fisher Exact Test Sir Ronald Aylmer Fisher (1890 –1962) Fisher’s exact test (named after Fisher for work he originally published in 1922, but also published in other ways after 28 possible absolute differences in averages between two groups 2/28 = 0.07 = probability of getting 2.5 from randomly sampling two groups from the same population

8. We can use resampling methods instead! Methods that take many samples from original data set in some specified way and evaluate the significance of the original based on these samples Approaches are nonparametric, because they generate a distribution from the data Are very flexible, and can allow for complicated designs Major variants: randomization, bootstrap, jackknife, Monte Carlo Enumerating all possible permutations becomes challenging for large data sets, several groups, and uneven group sizes. Table to the right shows the number of permutations needed for two equal sized groups BIOL 582 Resampling

9. Fisher’s exact test: a total enumeration of possible pairings of data (designed for contingency tables) Randomization can be used as a proxy (but for most any test statistic) Protocol Calculate observed statistic (e.g., ): E obs Reorder data set (i.e. randomly shuffle data) and recalculate statistic E rand Repeat for all possible combinations and generate distribution of possible statistics Percentage of E rand more extreme than E obs is significance level Note: E obs is treated as an iteration BIOL 582 Resampling E obs E rand E obs

10. Although it may not be feasible to consider all possible permutations, empirical and theoretical statistical research has shown that using MANY (e.g., > 1,000) random permutations works well. More specifically, the mean value obtained from the distribution of random values generally converges on the expected value for the NULL hypothesis. BIOL 582 Resampling E obs E rand E obs

11. What is a null hypothesis? In general, it’s the hypothesis that states “no relationship”, or “no difference”, which can be tested with evidence (data). If results from a hypothesis test support an alternative hypothesis (i.e., there is a relationship, or there is a difference), then the null hypothesis is rejected. The expected value for a null hypothesis states something about the population from which data are collected. It usually is presented as a difference equal to 0. BIOL 582 Null Hypotheses

12. Example: Imagine collecting a sample of 10 treefrogs from the Green River Biopreserve. The snout-to-vent length (SVL) is measured for each, and the sample mean is calculated. The sample mean is an estimate of the unknown, but existent, population mean. Go out and repeat the procedure. What is the expected difference in sample means between the two samples? BIOL 582 Null Hypotheses Answer: 0 Why: If the sample mean is an estimate of the population mean, and it’s accurate, then the difference in sample means should be the difference between the population mean and itself, which is 0!

13. BIOL 582 Null Hypotheses But there has to be some sort of expected error in sampling! A B Two samples collected from two populations Null Hypothesis (H 0 ): µ A = µ B ; or µ A - µ B = 0 H 0 implies that if the two population means are not different, sampling from both is tantamount to sampling from one. Important point: if two samples have parameter estimates that “behave” as if from one population, then randomly shuffling values between samples should indicate that the observed difference is not uncommon. If the sample parameter estimates are quite different, then the observed case should be rare compared to randomly shuffling values between samples. The best way to demonstrate this is with a physical example. Go to R

14. BIOL 582 The Language of Hypothesis Testing (1 – α)% Confidence intervals α is the accepted type I error rate (more later) E.g., a 95% Confidence interval is the distribution of values between 2.5 and 97.5 percentiles Imagine two confidence intervals from different populations (or samples): 25 +/- 10 35 +/- 5 Are the means different? Notice that the confidence intervals of the means overlap 1015202530354045 Question: do the populations have different means?

15. BIOL 582 The Language of Hypothesis Testing Hypothesis testing: Steps in Hypothesis Testing 1.A claim is made. 2.Evidence (sample data) is collected in order to test the claim. 3.The data are analyzed in order to support or refute the claim. Hypothesis testing is a procedure, based on sample evidence and probability, used to test claims regarding a characteristic of one or more populations. The null hypothesis, denoted H 0, is a statement to be tested. The null hypothesis is assumed true until evidence indicates otherwise. It is a statement regarding the value of a population parameter. The alternative hypothesis, denoted H a, is a claim to be tested. We are trying to find evidence for the alternative hypothesis. It is a claim about a population parameter.

16. BIOL 582 The Language of Hypothesis Testing Hypothesis testing: There are several ways to test a null hypothesis: 1. Equal hypothesis versus not equal hypothesis (two-sided test) H 0 : parameter = some value H a : parameter  some value 2. Equal hypothesis versus less than (one-sided test) H 0 : parameter = some value H a : parameter < some value 3. Equal hypothesis versus greater than (one-sided test) H 0 : parameter = some value H a : parameter > some value

17. BIOL 582 The Language of Hypothesis Testing Hypothesis testing: When we test a hypothesis, there are four possible outcomes: Four Outcomes from Hypothesis Testing 1.We reject the null when in fact the alternative is true. This decision would be correct. 2.We fail to reject the null when in fact the null is true. This decision would be correct. 3.We reject the null when in fact the null is true. This decision would be incorrect. This type of error is called a Type I error. 4.We fail to reject the null when in fact the alternative is true. This decision would be incorrect. This type of error is called a Type II error. Reality Conclusion H 0 is True H 0 is False Do not reject H 0 Correct ConclusionType II Error Reject H 0 Type I ErrorCorrect Conclusion

18. BIOL 582 The Language of Hypothesis Testing Hypothesis testing: Think about the Judicial Process: A person is innocent until proven guilty…. Thus, H 0 : defendant is innocent H a : defendant is guilty We have four possible outcomes: 1.The defendant is innocent and found to be innocent (i.e., null is true and the hypothesis test did not reject the null hypothesis). 2.The defendant is guilty and found to be guilty (i.e., null is false and the hypothesis test rejects the null hypothesis). 3.The defendant is really innocent but found to be guilty and sent to jail (Type I error). 4.The defendant is really guilty but found to be innocent and set free (Type II error).

19. BIOL 582 The Language of Hypothesis Testing Hypothesis testing: We may want to know the probability of making a Type I or Type II error.  : The probability of making a Type I error (called the level of significance)*  : The probability of making a Type II error. *  can be associated with empirical percentiles or associated with the tails of a theoretical distributions, such as standard normal or t -distribution. In both cases, it represents the probability of finding a value lower or higher than the confidence limits we describe (e.g.,  = 0.10 means we assume a 10% probability of finding a value outside of our confidence limits). In hypothesis testing,  is the probability of finding a value outside a range that is associated with the null hypothesis.

20. BIOL 582 The Language of Hypothesis Testing What kind of hypothesis tests can we learn? … we could systematically go through a bunch and learn them well, and learn in what situations to use them…. BORING! We will instead learn methods important to us (like linear models) and consider how to test hypotheses that are important

21. BIOL 582 Testing a Hypothesis Regardless, the following steps are useful to remember in hypothesis tests 1.State the null and alternative hypotheses. 2.Select the level of significance, . 3.Calculate the test statistic (e.g., Z value, t statistic, SS, means contrast, Λ 2 ) 4.Compare either 1.The critical value of the test statistic with the observed test statistic. 2.The observed statistic to a distribution of random statistics 5.State the conclusion. If the probability of finding a random statistic that exceeds the value of a test statistic associated with , is lower than , one rejects the null hypothesis In other words, if a chance outcome is rarer than our comfort level, we believe the null hypothesis is falsified.

22. BIOL 582 Testing a Hypothesis Now, let’s play with some data and see how this works. As we do, either on a notepad, or a document on your computer screen, always provide the following information for any test: H 0 : H a : α: Reject H 0 :? Conclusion: