7/2/2015Basics of Significance Testing1 Chapter 15 Tests of Significance: The Basics

7/2/2015Basics of Significance Testing2 Significance Testing Also called “hypothesis testing” Objective: to test a claim about parameter μ Procedure: A.State hypotheses H 0 and H a B.Calculate test statistic C.Convert test statistic to P-value and interpret D.Consider significance level (optional)

Basic Biostat9: Basics of Hypothesis Testing3 Hypotheses H 0 (null hypothesis) claims “no difference” H a (alternative hypothesis) contradicts the null Example: We test whether a population gained weight on average… H 0 : no average weight gain in population H a: H 0 is wrong (i.e., “weight gain”) Next  collect data  quantify the extent to which the data provides evidence against H 0

7/2/2015Basics of Significance Testing4 One-Sample Test of Mean To test a single mean, the null hypothesis is H 0 : μ = μ 0, where μ 0 represents the “null value” ( null value comes from the research question, not from data!) The alternative hypothesis can take these forms: H a : μ > μ 0 (one-sided to right) or H a : μ < μ 0 (one-side to left) or H a : μ ≠ μ 0 (two-sided) For the weight gain illustrative example: H 0 : μ = 0 H a : μ > 0 (one-sided) or H a : μ ≠ μ 0 (two-sided) Note: μ 0 = 0 in this example

7/2/2015Basics of Significance Testing5 Illustrative Example: Weight Gain Let X ≡ weight gain X ~N(μ, σ = 1), the value of μ unknown Under H 0, μ = 0 Take SRS of n = 10 σ x-bar = 1 / √(10) = Thus, under H 0 x-bar~N(0, 0.316) Figure: Two possible xbars when H 0 true

7/2/2015Basics of Significance Testing6 Take an SRS of size n from a Normal population. Population σ is known. Test H 0 : μ = μ 0 with: One-Sample z Statistic For “weight gain” data, x-bar = 1.02, n = 10, and σ = 1

7/2/2015Basics of Significance Testing7 P-value P-value ≡ the probability the test statistic would take a value as extreme or more extreme than observed test statistic, when H 0 is true Smaller-and-smaller P-values → stronger-and- stronger evidence against H 0 Conventions for interpretation P >.10  evidence against H 0 not significant.05 < P ≤.10  evidence marginally significant.01 < P ≤.05  evidence against H 0 significant P ≤.01  evidence against H 0 very significant

7/2/2015Basics of Significance Testing8 P-Value Convert z statistics to P-value : For H a : μ  > μ 0 P = Pr(Z > z stat ) = right-tail beyond z stat For H a : μ  < μ 0 P = Pr(Z < z stat ) = left tail beyond z stat For H a : μ  μ 0 P = 2 × one-tailed P-value

7/2/2015Basics of Significance Testing9 Illustrative Example z statistic = 3.23 One-sided P = P(Z > 3.23) = 1− = Highly significant evidence against H 0

7/2/2015Basics of Significance Testing10 α ≡ threshold for “significance” We set α For example, if we choose α = 0.05, we require evidence so strong that it would occur no more than 5% of the time when H 0 is true Decision rule P ≤ α  statistically significant evidence P > α  nonsignificant evidence For example, if we set α = 0.01, a P-value of is considered significant Significance Level

7/2/2015Basics of Significance Testing11 Summary

7/2/2015Basics of Significance Testing12 Illustrative Example: Two-sided test 1. Hypotheses: H 0 : μ = 0 against H a : μ ≠ 0 2. Test Statistic: 3. P-value: P = 2 × Pr(Z > 3.23) = 2 × = Conclude  highly significant evidence against H 0

7/2/2015Basics of Significance Testing13 Relation Between Tests and CIs For two-sided tests, significant results at the α- level  μ 0 will fall outside (1–α)100% CI When α =.05  (1–α)100% = (1–.05)100% = 95% confidence When α =.01, (1–α)100% = (1–.01)100% = 99% confidence Recall that we tested H 0 : μ = 0 and found a two- sided P = Since this is significant at α =.05, we expect “0” to fall outside that 95% confidence interval … continued …

7/2/2015Basics of Significance Testing14 Relation Between Tests and CIs Recall: xbar = 1.02, n = 10, σ = 1. Therefore, a 95% CI for μ = Since 0 falls outside this 95% CI  the test of H 0 : μ = 0 is significant at α =.05

7/2/2015Basics of Significance Testing15 Example II: Job Satisfaction The null hypothesis “no average difference” in the population of workers. The alternative hypothesis is “there is an average difference in scores” in the population. H 0 :  = 0H a :  ≠ 0 This is a two-sided test because we are interested in differences in either direction. Does the job satisfaction of assembly workers differ when their work is machine-paced rather than self-paced? A matched pairs study was performed on a sample of workers. Workers’ satisfaction was assessed in each setting. The response variable is the difference in satisfaction scores, self-paced minus machine-paced.

7/2/2015Basics of Significance Testing16 Illustrative Example II  Job satisfaction scores follow a Normal distribution with standard deviation  = 60. u Data from 18 workers gives a sample mean difference score of 17. u Test H 0 : µ = 0 against H a : µ ≠ 0 with

7/2/2015Basics of Significance Testing17 Illustrative Example II Two-sided P-value = Pr( Z 1.20 ) = 2 × Pr (Z > 1.20) = (2)(0.1151) = Conclude: chance we would see results this extreme when H 0 is true  evidence against H 0 not strong (not significant)

7/2/2015Basics of Significance Testing18 Example II: Conf Interval Method This 90% CI includes 0. Therefore, it is plausible that the true value of  is 0  H 0 : µ = 0 cannot be rejected at α = Studying Job Satisfaction A 90% CI for μ is