Significance Tests: THE BASICS Could it happen by chance alone?

Statistical Inference Confidence Intervals—Use when you want to estimate a population parameter Confidence Intervals—Use when you want to estimate a population parameter Significance Tests—Use when you want to assess the evidence provided by data about some claim concerning a population Significance Tests—Use when you want to assess the evidence provided by data about some claim concerning a population –AN OUTCOME THAT WOULD RARELY HAPPEN BY CHANCE IF A CLAIM WERE TRUE IS GOOD EVIDENCE THAT THE CLAIM IS NOT TRUE

Overview of a Significance Test A test of significance is intended to assess the evidence provided by data against a null hypothesis H 0 in favor of an alternate hypothesis H a. A test of significance is intended to assess the evidence provided by data against a null hypothesis H 0 in favor of an alternate hypothesis H a. The statement being tested in a test of significance is called the null hypothesis. Usually the null hypothesis is a statement of “no effect” or “no difference.” The statement being tested in a test of significance is called the null hypothesis. Usually the null hypothesis is a statement of “no effect” or “no difference.” A one-sided alternate hypothesis exists when we are interested only in deviations from the null hypothesis in one direction A one-sided alternate hypothesis exists when we are interested only in deviations from the null hypothesis in one direction H 0 :  =0 H a :  >0 (or  0 (or  <0) If the problem does not specify the direction of the difference, the alternate hypothesis is two-sided If the problem does not specify the direction of the difference, the alternate hypothesis is two-sided H 0 :  =0 H a :  ≠0

HYPOTHESES NOTE: Hypotheses ALWAYS refer to a population parameter, not a sample statistic. NOTE: Hypotheses ALWAYS refer to a population parameter, not a sample statistic. The alternative hypothesis should express the hopes or suspicions we have BEFORE we see the data. Don’t “cheat” by looking at the data first. The alternative hypothesis should express the hopes or suspicions we have BEFORE we see the data. Don’t “cheat” by looking at the data first.

CONDITIONS These should look the same as in the last chapter (for confidence intervals) These should look the same as in the last chapter (for confidence intervals) –Random  Data is from an SRS or from a randomized experiment –Normal  For means—population distribution is Normal or you have a large sample size (n≥30) to ensure a Normal sampling distribution for the sample mean  For proportions—np≥10 and n(1-p)≥10 (meaning the sample is large enough to ensure a Normal sampling distribution for –Independent  Either you are sampling with replacement or you have a population at least 10 times as big as the sample to make using the formula for st. dev. okay.

CAUTION Be sure to check that the conditions for running a significance test for the population mean are satisfied before you perform any calculations. Be sure to check that the conditions for running a significance test for the population mean are satisfied before you perform any calculations.

Test Statistic A test statistic comes from sample data and is used to make decisions in a significance test A test statistic comes from sample data and is used to make decisions in a significance test –Compare sample statistic to hypothesized parameter –Values far from parameter give evidence against the null hypothesis (H 0 ) –Standardize your sample statistic to obtain your TEST STATISTIC

P-values & statistical significance The probability (computed assuming H 0 is true) that the test statistic would take a value as extreme or more extreme than that actually observed is called the P-value of the test. The smaller the P-value, the stronger the evidence against the null hypothesis provided by the data. The probability (computed assuming H 0 is true) that the test statistic would take a value as extreme or more extreme than that actually observed is called the P-value of the test. The smaller the P-value, the stronger the evidence against the null hypothesis provided by the data. “Significant” in the statistical sense doesn’t mean “important”. It means simply “not likely to happen just by chance.” “Significant” in the statistical sense doesn’t mean “important”. It means simply “not likely to happen just by chance.” The significance level α is the decisive value of the P-value. It makes “not likely” more exact. The significance level α is the decisive value of the P-value. It makes “not likely” more exact. If the P-value is as small or smaller than α, we say that the data is statistically significant at level α. If the P-value is as small or smaller than α, we say that the data is statistically significant at level α.

INFERENCE TOOLBOX (p 705) 1—PARAMETER—Identify the population of interest and the parameter you want to draw a conclusion about. STATE YOUR HYPOTHESES! 1—PARAMETER—Identify the population of interest and the parameter you want to draw a conclusion about. STATE YOUR HYPOTHESES! 2—CONDITIONS—Choose the appropriate inference procedure. VERIFY conditions (Random, Normal, Independent) before using it. 2—CONDITIONS—Choose the appropriate inference procedure. VERIFY conditions (Random, Normal, Independent) before using it. 3—CALCULATIONS—If the conditions are met, carry out the inference procedure. 3—CALCULATIONS—If the conditions are met, carry out the inference procedure. 4—INTERPRETATION—Interpret your results in the context of the problem. CONCLUSION, CONNECTION, CONTEXT(meaning that our conclusion about the parameter connects to our work in part 3 and includes appropriate context) 4—INTERPRETATION—Interpret your results in the context of the problem. CONCLUSION, CONNECTION, CONTEXT(meaning that our conclusion about the parameter connects to our work in part 3 and includes appropriate context) Steps for completing a SIGNIFICANCE TEST: DO YOU REMEMBER WHAT THE STEPS ARE???

Step 1—PARAMETER Read through the problem and determine what we hope to show through our test. Read through the problem and determine what we hope to show through our test. Our null hypothesis is that no change has occurred or that no difference is evident. Our null hypothesis is that no change has occurred or that no difference is evident. Our alternative hypothesis can be either one or two sided. Our alternative hypothesis can be either one or two sided. Be certain to use appropriate symbols and also write them out in words. Be certain to use appropriate symbols and also write them out in words.

Step 2—CONDITIONS Based on the given information, determine which test should be used. Name the procedure. Based on the given information, determine which test should be used. Name the procedure. State the conditions. State the conditions. Verify (through discussion) whether the conditions have been met. For any assumptions that seem unsafe to verify as met, explain why. Verify (through discussion) whether the conditions have been met. For any assumptions that seem unsafe to verify as met, explain why. Remember, if data is given, graph it to help facilitate this discussion Remember, if data is given, graph it to help facilitate this discussion For each procedure there are several things that we are assuming are true that allow these procedures to produce meaningful results. For each procedure there are several things that we are assuming are true that allow these procedures to produce meaningful results.

Step 3—CALCULATIONS First write out the formula for the test statistic, report its value, mark the value on the curve. First write out the formula for the test statistic, report its value, mark the value on the curve. Sketch the density curve as clearly as possible out to three standard deviations on each side. Sketch the density curve as clearly as possible out to three standard deviations on each side. Mark the null hypothesis and sample statistic clearly on the curve. Mark the null hypothesis and sample statistic clearly on the curve. Calculate and report the P-value Calculate and report the P-value Shade the appropriate region of the curve. Shade the appropriate region of the curve. Report other values of importance (standard deviation, df, critical value, etc.) Report other values of importance (standard deviation, df, critical value, etc.)

Step 4—INTERPRETATION There are really two parts to this step: decision & conclusion. TWO UNIQUE SENTENCES. There are really two parts to this step: decision & conclusion. TWO UNIQUE SENTENCES. Based on the P-value, make a decision. Will you reject H 0 or fail to reject H 0. Based on the P-value, make a decision. Will you reject H 0 or fail to reject H 0. If there is a predetermined significance level, then make reference to this as part of your decision. If not, interpret the P-value appropriately. If there is a predetermined significance level, then make reference to this as part of your decision. If not, interpret the P-value appropriately. Now that you have made a decision, state a conclusion IN THE CONTEXT of the problem. Now that you have made a decision, state a conclusion IN THE CONTEXT of the problem. This does not need to, and probably should not, have statistical terminology involved. DO NOT use the word “prove” in this statement. This does not need to, and probably should not, have statistical terminology involved. DO NOT use the word “prove” in this statement.

Example 1 Your buddy (Jake) claims to be an A student (meaning he has a 90 average). You don’t know all of his grades but based on what you have seen you think this claim is an overstatement. You took a simple random sample of his grades and recorded them. They are: 92, 87, 86, 90, 80, 91. You also know that all his grades in the class have a standard deviation of 3.5. Your buddy (Jake) claims to be an A student (meaning he has a 90 average). You don’t know all of his grades but based on what you have seen you think this claim is an overstatement. You took a simple random sample of his grades and recorded them. They are: 92, 87, 86, 90, 80, 91. You also know that all his grades in the class have a standard deviation of 3.5.

Step 1 We want to determine whether Jake is accurate in his measure of his course grade. We want to determine whether Jake is accurate in his measure of his course grade. Our null hypothesis is that Jake has a course average of 90. Our null hypothesis is that Jake has a course average of 90. Our alternative hypothesis is that Jake’s course average is below a 90. Our alternative hypothesis is that Jake’s course average is below a 90. H 0 :  = 90 H 0 :  = 90 H a :  < 90 H a :  < 90

Step 2 Since we know the population standard deviation we will be performing a z-test of significance. (NOTE-in practice, we rarely know sigma) Since we know the population standard deviation we will be performing a z-test of significance. (NOTE-in practice, we rarely know sigma) We were told that our selection of grades was an SRS of Jake’s scores. We were told that our selection of grades was an SRS of Jake’s scores. The box plot shows moderate left skewness. Our sample is not large so we must assume that the population of all of Jake’s grades are approximately normal in distribution in order for our sampling distribution to be approximately normal. Using the IQR(1.5) method for determining outliers we see that there are no outliers in this sample of grades. The box plot shows moderate left skewness. Our sample is not large so we must assume that the population of all of Jake’s grades are approximately normal in distribution in order for our sampling distribution to be approximately normal. Using the IQR(1.5) method for determining outliers we see that there are no outliers in this sample of grades. Provided Jake has at least 60 overall grades, we are safe assuming independence and using the necessary formula for standard deviation. Provided Jake has at least 60 overall grades, we are safe assuming independence and using the necessary formula for standard deviation.

Step 3 A curve should be drawn, labeled, and shaded. A curve should be drawn, labeled, and shaded. You can use the formula to calculate your z test statistic for this problem You can use the formula to calculate your z test statistic for this problem  In this case z =  In this case z = Mark this on your sketch. Mark this on your sketch. Based on our calculations the P-value is Based on our calculations the P-value is , σ=3.5, n=6, σ=3.5, n=6

Step 4 Since there is no predetermined level of significance if we are seeking to make a decision, this could be argued either way. If Jake were correct about being an A student, we would only get a sample of grades with an average this low in roughly 5.1% of all samples. Since there is no predetermined level of significance if we are seeking to make a decision, this could be argued either way. If Jake were correct about being an A student, we would only get a sample of grades with an average this low in roughly 5.1% of all samples. There is not overwhelming evidence against H 0, however, this is enough to convince me that H 0 can be rejected. There is not overwhelming evidence against H 0, however, this is enough to convince me that H 0 can be rejected. Our evidence may not be strong enough to convince Jake that he is wrong. However, based on this evidence, I do not believe Jake is accurate about his average being a 90. It doesn’t appear that Jake is the A student he claims to be. Our evidence may not be strong enough to convince Jake that he is wrong. However, based on this evidence, I do not believe Jake is accurate about his average being a 90. It doesn’t appear that Jake is the A student he claims to be.

WARNINGS Tests of significance assess evidence against H 0 If the evidence is strong, reject H 0 in favor of H a Failure to find evidence against H 0 means only that data are consistent with H 0, not that we have clear evidence that H 0 is true α If you are going to make a decision based on statistical significance, then the significance level α should be stated before the data are produced.