Hypothesis Testing III 2/15/12 Statistical significance Errors Power Significance and sample size Section 4.3 Professor Kari Lock Morgan Duke University

Project 1 Proposal (due today, 5pm) Project 1 Proposal Homework 4 (due Monday) Homework 4 If you turn in your HW4 by 5pm on Friday (either slide it under the door of Old Chem 216 or it to your TA), it will be graded by class on Monday NO LATE HOMEWORK ACCEPTED! Study/prepare for Exam 1! To Do

Exam 1: In-class portion: Wednesday, 2/22 Lab portion: Thursday, 2/23 In-class portion: (75%) Open only to a calculator and one double sided page of notes prepared by you Emphasis on conceptual understanding Lab portion : (25%) Open to everything except communication of any form with other humans Emphasis on actually analyzing data Exam 1

Last year’s in-class and lab midterms, with solutions, are available on the course website (under documents) Full solutions to ALL the essential synthesis and review problems from Units 1 and 2 are available on the course website Doing problems is the key to success!!! Practice

Work lots of practice problems! Take last year’s exams under realistic conditions (time yourself, do it all before looking at the solutions, etc.) Prepare a good cheat sheet and use it when working problems Read the corresponding sections in the book if there are concepts you are still confused about Keys to In-Class Exam Success

Primarily, make sure you know how to summarize, visualize, create an interval, and conduct a test for any one variable or relationship between two variables. Beyond that, make sure you are comfortable with the content from the labs Open-book does NOT mean you don’t have to study. You will not have time to look up every command you need during the exam. Keys to Lab Exam Success

You have LOTS of opportunities for help! Today, 3 – 5pm (Prof Morgan) Friday, 1 – 3 pm (Prof Morgan Sunday, 5 – 7 pm (Jessica) Sunday, 7 – 9 pm (Michael) Monday, 3 – 4 pm (Prof Morgan) Monday, 4 – 6 pm (Christine) Tuesday, 3 – 6 pm (Prof Morgan) Tuesday, 6 – 8 pm (Yue) (My office hours next week have been moved to Monday and Tuesday to answer questions before the exam) Office Hours before Exam

Statistical Conclusions Strength of evidence against H 0 : Formal decision of hypothesis test, based on  = 0.05 :

Resveratrol, an ingredient in red wine and grapes, has been shown to promote weight loss in rodents, and has recently been investigated in primates (specifically, the Grey Mouse Lemur). A sample of lemurs had various measurements taken before and after receiving resveratrol supplementation for 4 weeks Red Wine and Weight Loss BioMed Central (2010, June 22). “Lemurs lose weight with ‘life-extending’ supplement resveratrol. Science Daily.

In the test to see if the mean resting metabolic rate is higher after treatment, the p-value is Using  = 0.05, is this difference statistically significant? (should we reject H 0 : no difference?) (a) Yes (b) No Red Wine and Weight Loss

In the test to see if the mean body mass is lower after treatment, the p-value is Using  = 0.05, is this difference statistically significant? (should we reject H 0 : no difference?) (a) Yes (b) No Red Wine and Weight Loss

In the test to see if locomotor activity changes after treatment, the p-value is Using  = 0.05, is this difference statistically significant? (should we reject H 0 : no difference?) (a) Yes (b) No Red Wine and Weight Loss

In the test to see if mean food intake changes after treatment, the p-value is Using  = 0.05, is this difference statistically significant? (should we reject H 0 : no difference?) (a) Yes (b) No Red Wine and Weight Loss

Suppose many researchers around the world are all investigating the same question of interest. If the null hypothesis is true, using  = 0.05, what proportion of hypothesis tests will incorrectly reject the null? a)None b)95% c)5% d)It depends Formal Decisions

There are four possibilities: Errors Reject H 0 Do not reject H 0 H 0 true H 0 false TYPE I ERROR TYPE II ERROR Truth Decision

In the test to see if resveratrol is associated with food intake, the p-value is If resveratrol is not associated with food intake, a Type I Error would have been made In the test to see if resveratrol is associated with locomotor activity, the p-value is If resveratrol is associated with locomotor activity, a Type II Error would have been made Red Wine and Weight Loss

Usually, we have no way of knowing whether an error has been made, without doing another study Analogously, we have no way to knowing whether our confidence interval actually contains the true parameter Errors

If the null hypothesis is true, what is the probability of making a Type I error? a)0 b)  c)1 –  d)It depends Errors

Why would you use a smaller significance level, like  = 0.01? Why would you use a larger significance level, like  = 0.10? Significance Level

A person is innocent until proven guilty. Evidence must be beyond the shadow of a doubt. Types of mistakes in a verdict? Convict an innocent Release a guilty HoHo HaHa  Type I error Type II error Analogy to Law

If the alternative hypothesis is true, what is the probability of making a Type II error? a)0 b)  c)1 –  d)It depends Errors

The power of a hypothesis test is the probability of correctly detecting a significant effect, is there is one (correctly rejecting the null hypothesis when it is false) Power

There are four possibilities: Errors Reject H 0 Do not reject H 0 H 0 true H 0 false TYPE I ERROR TYPE II ERROR Truth Decision

What factors influence the power of a test? 1. Sample size 2. True value or effect size 3. Variability of values (standard deviation) Power

If you want to increase the power of your test, what can you do? a)Increase the sample size b)Make the true value farther from the null value c)Decrease the standard deviation of your variables d)Any of the above

Significance and Sample Size

The smaller the sample size, the (a) smaller (b) larger the chance of a Type II error (failing to reject the null hypothesis, even when it is false).

Significance and Sample Size Just because you fail to get significant results, does NOT mean the null hypothesis is true This is particularly true for small sample sizes. Unless the truth is very far from the null value, it is hard to find significant results if the sample size is small. With small sample sizes, Type II Errors are very likely!

With small sample sizes, even large differences or effects may not be significant With large sample sizes, even a very small difference or effect can be significant A statistically significant result is not always practically significant, especially with large sample sizes Statistical vs Practical Significance

Example: Suppose a weight loss program recruits 10,000 people for a randomized experiment. A difference in average weight loss of only 0.5 lbs could be found to be statistically significant Suppose the experiment lasted for a year. Is a loss of ½ a pound practically significant? Statistical vs Practical Significance

Videogames and GPA If you get put with a roommate who brings a videogame to college, does that lower your GPA? What are the null and alternative hypotheses? a)H 0 : p v – p nv = 0, H a : p v – p nv < 0 b)H 0 : µ v – µ nv = 0, H a : µ v – µ nv < 0 c)H 0 : p v – p nv < 0, H a : p v – p nv = 0 d)H 0 : µ v – µ nv < 0, H a : µ v – µ nv = 0

If you get put with a roommate who brings a videogame to college, does that lower your GPA? A study at Berea college conducted this test and obtained a p-value of What does this mean? a)The probability that H 0 is true is b)The probability that H 0 is false is c)The probability of seeing a difference in mean GPA as extreme as that in the sample is d)If H 0 is true, the probability of seeing a difference in mean GPA as extreme as that in the sample is Source: Stinebrickner, R. and Stinebrickner, T.R. (2008). “The Causal Effect of Studying on Academic Performance,” The B.E. Journal of Economic Analysis & Policy: Vol. 8: Iss. 1 (Frontiers), Article 14. Videogames and GPA

In the study about roommates bringing videogames to college and GPA, the p-value is Using  = 0.05, what would you conclude? a) People assigned roommates who bring videogames have significantly lower GPAs b)People assigned roommates who bring videogames do not have lower GPAs c)Nothing Videogames and GPA

Based on this p-value, can you conclude that getting put with a roommate who brings a videogame to campus causes you to have a lower GPA, on average? a)Yes b)No Videogames and GPA

The p-value alone tells you whether there is a significant association between two variables, but NOT whether this is a causal association The data collection method tells you whether causal conclusions are possible, but not whether an association is significant If the study is a randomized experiment AND the p–value indicates statistically significant results, only then you can conclude that the explanatory variable has a causal effect on the response variable Significance and Causation

Roommates are assigned randomly at Berea college. Based on this knowledge and the p-value (0.036), can you conclude that getting put with a roommate who brings a videogame to campus causes you to have a lower GPA, on average? a)Yes b)No Videogames and GPA

The study also tested whether students who bring a videogame to college themselves have lower GPAs, on average. The p-value of this test is Using  = 0.05, what would you conclude? a) People who bring videogames have significantly lower GPAs b)People who bring videogames do not have lower GPAs c)In this study, the difference in average GPA between students who bring videogames and those who don’t is not statistically significant d)Nothing e)Either (c) or (d) Videogames and GPA

In making formal decisions, reject H 0 if the p- value is less than α, otherwise do not reject H 0 There are two types of errors that can be made in hypothesis testing: rejecting a true null or failing to reject a false null The larger your sample size, the higher your chance of finding a significant result, if one exists Summary