Power and Sample Size IF IF the null hypothesis H 0 : μ = μ 0 is true, then we should expect a random sample mean to lie in its “acceptance region” with probability 1 – α, the “confidence level.” That is, P(Accept H 0 | H 0 is true) = 1 – α. Therefore, we should expect a random sample mean to lie in its “rejection region” with probability α, the “significance level.” That is, P(Reject H 0 | H 0 is true) = α. 1   H 0 :  =  0 Acceptance Region for H 0 Rejection Region  /2 “Null Distribution” “Type 1 Error” μ 0 + z α/2 (σ /

Power and Sample Size 1   H 0 :  =  0 Acceptance Region for H 0 Rejection Region  /2 “Null Distribution” “Type 2 Error” μ 0 + z α/2 (σ / “Alternative Distribution” 1 –   H A : μ = μ 1 μ 1 – z  (σ / IF IF the null hypothesis H 0 : μ = μ 0 is false, then the “power” to correctly reject it in favor of a particular alternative H A : μ = μ 1 is P(Reject H 0 | H 0 is false) = 1 – . Thus, P(Accept H 0 | H 0 is false) = . Set them equal to each other, and solve for n …

N(0, 1) X ~ N( μ, σ )Normally-distributed population random variable, with unknown mean, but known standard deviation H 0 : μ = μ 0 Null Hypothesis value H A : μ = μ 1 Alternative Hypothesis specific value  significance level (or equivalently, confidence level 1 –  ) 1 –  power (or equivalently, Type 2 error rate  ) Then the minimum required sample size is: Given: Example: σ = 1.5 yrs, μ 0 = 25.4 yrs,  =.05 1    zz Suppose it is suspected that currently, μ 1 = 26 yrs. Want 90% power of correctly rejecting H 0 in favor of H A, if it is false  z.025 = 1.96  1 –  =.90   =.10  z.10 = 1.28  = |26 – 25.4| / 1.5 = 0.4 So… minimum sample size required is n  66 Want more power!

n  66 n  82 N(0, 1) X ~ N( μ, σ )Normally-distributed population random variable, with unknown mean, but known standard deviation H 0 : μ = μ 0 Null Hypothesis value H A : μ = μ 1 Alternative Hypothesis specific value  significance level (or equivalently, confidence level 1 –  ) 1 –  power (or equivalently, Type 2 error rate  ) Then the minimum required sample size is: Given: 1    zz Want 90% power of correctly rejecting H 0 in favor of H A, if it is false  1 –  =.90 So… minimum sample size required is Want 95% power of correctly rejecting H 0 in favor of H A, if it is false  1 –  =.95   =.10   =.05  z.10 = 1.28  z.05 = Example: σ = 1.5 yrs, μ 0 = 25.4 yrs,  =.05 Suppose it is suspected that currently, μ 1 = 26 yrs.  z.025 = 1.96  = |26 – 25.4| / 1.5 = 0.4 Change μ 1

 = |26 – 25.4| / 1.5 = 0.4  = |25.7 – 25.4| / 1.5 = 0.2 Suppose it is suspected that currently, μ 1 = 26 yrs.Suppose it is suspected that currently, μ 1 = 25.7 yrs. n  82 N(0, 1) X ~ N( μ, σ )Normally-distributed population random variable, with unknown mean, but known standard deviation H 0 : μ = μ 0 Null Hypothesis value H A : μ = μ 1 Alternative Hypothesis specific value  significance level (or equivalently, confidence level 1 –  ) 1 –  power (or equivalently, Type 2 error rate  ) Then the minimum required sample size is: Given: Example: σ = 1.5 yrs, μ 0 = 25.4 yrs,  =.05 1    zz  z.025 = 1.96 So… minimum sample size required is Want 95% power of correctly rejecting H 0 in favor of H A, if it is false  1 –  =.95   =.05  z.05 = n  325

N(0, 1) X ~ N( μ, σ )Normally-distributed population random variable, with unknown mean, but known standard deviation H 0 : μ = μ 0 Null Hypothesis value H A : μ = μ 1 Alternative Hypothesis specific value  significance level (or equivalently, confidence level 1 –  ) 1 –  power (or equivalently, Type 2 error rate  ) Then the minimum required sample size is: Given: 1    zz With n = 400, how much power exists to correctly reject H 0 in favor of H A, if it is false? Power = 1 –  = = , i.e., 98% Example: σ = 1.5 yrs, μ 0 = 25.4 yrs,  =.05 Suppose it is suspected that currently, μ 1 = 25.7 yrs.  z.025 = 1.96

But this introduces additional variability from one sample to another… PROBLEM! X ~ N( μ, σ )Normally-distributed population random variable, with unknown mean, but known standard deviation H 0 : μ = μ 0 Null Hypothesis H A : μ ≠ μ 0 Alternative Hypothesis (2-sided)  significance level (or equivalently, confidence level 1 –  ) n sample size From this, we obtain… “standard error” s.e. sample mean sample standard deviation …with which to test the null hypothesis (via CI, AR, p-value). In practice however, it is far more common that the true population standard deviation σ is unknown. So we must estimate it from the sample! Given: x 1, x 2,…, x n x 1, x 2,…, x n Recall that (estimate)

X ~ N( μ, σ )Normally-distributed population random variable, with unknown mean, but known standard deviation H 0 : μ = μ 0 Null Hypothesis H A : μ ≠ μ 0 Alternative Hypothesis (2-sided)  significance level (or equivalently, confidence level 1 –  ) n sample size From this, we obtain… “standard error” s.e. sample mean sample standard deviation …with which to test the null hypothesis (via CI, AR, p-value). SOLUTION: follows a different sampling distribution from before. Given: x 1, x 2,…, x n But this introduces additional variability from one sample to another… PROBLEM! (estimate)

William S. Gossett ( ) … is actually a family of distributions, indexed by the degrees of freedom, labeled t df. As the sample size n gets large, t df converges to the standard normal distribution Z ~ N(0, 1). So the T-test is especially useful when n < 30. t1t1 t2t2 t3t3 t 10 Z ~ N(0, 1)

William S. Gossett ( ) … is actually a family of distributions, indexed by the degrees of freedom, labeled t df. As the sample size n gets large, t df converges to the standard normal distribution Z ~ N(0, 1). So the T-test is especially useful when n < 30. Z ~ N(0, 1) t4t

Lecture Notes Appendix… or… qt(.025, 4, lower.tail = F) [1]

William S. Gossett ( ) … is actually a family of distributions, indexed by the degrees of freedom, labeled t df. Because any t-distribution has heavier tails than the Z-distribution, it follows that for the same right-tailed area value, t-score > z-score. Z ~ N(0, 1) t4t

X = Age at first birth ~ N( μ, σ ) H 0 : μ = 25.4 yrs Null Hypothesis H A : μ ≠ 25.4 yrs Alternative Hypothesis Given: Previously… σ = 1.5 yrs, n = 400, statistically significant at  =.05 Example: n = 16, s = 1.22 yrs Now suppose that σ is unknown, and n < 30. standard error (estimate) =.025 critical value = t 15,.025

Lecture Notes Appendix…

p-value = X = Age at first birth ~ N( μ, σ ) H 0 : μ = 25.4 yrs Null Hypothesis H A : μ ≠ 25.4 yrs Alternative Hypothesis Given: Previously… σ = 1.5 yrs, n = 400, statistically significant at  =.05 Example: n = 16, s = 1.22 yrs Now suppose that σ is unknown, and n < 30. standard error (estimate) = 95% Confidence Interval =.025 critical value = t 15, % margin of error = (2.131)(0.305 yrs) = 0.65 yrs (25.9 – 0.65, ) = (25.25, 26.55) yrs  Test Statistic: = 2.131

Lecture Notes Appendix…

X = Age at first birth ~ N( μ, σ ) H 0 : μ = 25.4 yrs Null Hypothesis H A : μ ≠ 25.4 yrs Alternative Hypothesis Given: Example: n = 16, s = 1.22 yrs Now suppose that σ is unknown, and n < 30. standard error (estimate) = 95% Confidence Interval =.025 critical value = t 15,.025 = % margin of error = (2.131)(0.305 yrs) = 0.65 yrs p-value = = 2 (between.05 and.10) The 95% CI does contain the null value μ = The p-value is between.10 and.20, i.e., >.05. (Note: The R command 2 * pt(1.639, 15, lower.tail = F) gives the exact p-value as.122.) Not statistically significant; small n gives low power! = between.10 and.20. Previously… σ = 1.5 yrs, n = 400, statistically significant at  =.05 (25.9 – 0.65, ) = (25.25, 26.55) yrs

Assuming X ~ N( , σ), test H 0 :  =  0 vs. H A :  ≠  0, at level α… Lecture Notes Appendix A3.3…Lecture Notes Appendix A3.3… (click for details on this section) If the population variance  2 is known, then use it with the Z-distribution, for any n. If the population variance  2 is unknown, then estimate it by the sample variance s 2, and use: either T-distribution (more accurate), or the Z-distribution (easier), if n  30, T-distribution only, if n < 30. To summarize… Assuming X ~ N( , σ) ALSO SEE PAGE

Assuming X ~ N( , σ), test H 0 :  =  0 vs. H A :  ≠  0, at level α… If the population variance  2 is known, then use it with the Z-distribution, for any n. If the population variance  2 is unknown, then estimate it by the sample variance s 2, and use: either T-distribution (more accurate), or the Z-distribution (easier), if n  30, T-distribution only, if n < 30. To summarize… Lecture Notes Appendix A3.3…Lecture Notes Appendix A3.3… (click for details on this section) Assuming X ~ N( , σ) ALSO SEE PAGE

Assuming X ~ N( , σ) And what do we do if it’s not, or we can’t tell? Z ~ N(0, 1) IF our data approximates a bell curve, then its quantiles should “line up” with those of N(0, 1). How do we check that this assumption is reasonable, when all we have is a sample?

Z ~ N(0, 1) Assuming X ~ N( , σ) How do we check that this assumption is reasonable, when all we have is a sample? And what do we do if it’s not, or we can’t tell? IF our data approximates a bell curve, then its quantiles should “line up” with those of N(0, 1). Sample quantiles Q-Q plot Normal scores plot Normal probability plot

Assuming X ~ N( , σ) How do we check that this assumption is reasonable, when all we have is a sample? And what do we do if it’s not, or we can’t tell? IF our data approximates a bell curve, then its quantiles should “line up” with those of N(0, 1). Q-Q plot Normal scores plot Normal probability plot (R uses a slight variation to generate quantiles…) qqnorm(mysample)

Assuming X ~ N( , σ) How do we check that this assumption is reasonable, when all we have is a sample? IF our data approximates a bell curve, then its quantiles should “line up” with those of N(0, 1). qqnorm(mysample) (R uses a slight variation to generate quantiles…) Q-Q plot Normal scores plot Normal probability plot Formal statistical tests exist; see notes. And what do we do if it’s not, or we can’t tell?

x = rchisq(1000, 15) hist(x) y = log(x) hist(y) Assuming X ~ N( , σ) How do we check that this assumption is reasonable, when all we have is a sample? And what do we do if it’s not, or we can’t tell?  Use a mathematical “transformation” of the data (e.g., log, square root,…). X is said to be “log-normal.”

 Use a mathematical “transformation” of the data (e.g., log, square root,…). Assuming X ~ N( , σ) How do we check that this assumption is reasonable, when all we have is a sample? And what do we do if it’s not, or we can’t tell? qqnorm(x, pch = 19, cex =.5) qqline(x) qqnorm(y, pch = 19, cex =.5) qqline(y)

Assuming X ~ N( , σ) How do we check that this assumption is reasonable, when all we have is a sample? And what do we do if it’s not, or we can’t tell?  Use a mathematical “transformation” of the data (e.g., log, square root,…).  Use a “nonparametric test” (e.g., Sign Test, Wilcoxon Signed Rank Test). These tests make no assumptions on the underlying population distribution! = Mann-Whitney Test Based on “ranks” of the ordered data; tedious by hand… Has less power than Z-test or T-test (when appropriate)… but not bad. In R, see ?wilcox.test for details…. SEE LECTURE NOTES, PAGE FOR FLOWCHART OF METHODS

See… _Statistical_Inference/HYPOTHESIS_TESTING_SUMMARY.pdf