1. Significance Tests Hypothesis - Statement Regarding a Characteristic of a Variable or set of variables. Corresponds to population(s) –Majority of registered voters favor health care reform –Average salary progressions differ for male executives whose spouses work than for those whose spouses “stay at home” Significance Test - Means of using sample statistics (and their sampling distributions) to compare their observed values with hypothesized value of corresponding parameter(s)

2. Elements of Significance Test (I) Assumptions –Data Type: Quantitative vs. Qualitative –Population Distribution: Some methods assume normal –Sampling Plan: Simple Random Sampling –Sample Size: Some methods have sample size requirements for validity Hypotheses –Null Hypothesis (H 0 ): A statement that parameter(s) take on specific value(s) (Often: “No effect”) –Alternative Hypothesis (H a ): A statement contradicting the parameter value(s) in the null hypothesis

3. Elements of Significance Test (II) Test Statistic: Quantity based on the sample data to test the null hypothesis. Typically is based on a sample statistic, parameter value under H 0, and the standard error. P-value (P): The probability that we would obtain a test statistic at least as contradictory to the null hypothesis as our computed test statistic, if the null hypothesis is true. –Small P-values mean the sample data are not consistent with the parameter value(s) under H 0

4. Elements of Significance Test (III) Conclusion (Optional) –If the P-value is sufficiently small, we reject H 0 in favor of H a. The most widely accepted minimum level is 0.05, and the test is said to be significant at the.05 level. –If the P-value is not sufficiently small, we fail to reject (but not necessarily accept) the null hypothesis. –Process is analogous to American judicial system H 0 : Defendant is innocent H a : Defendant is guilty

5. Significance Test for Mean (Large-Sample) Assumptions: Random sample with n  30, quantitative variable Null Hypothesis: H 0 :  =  0 (typically no effect or change from standard) Alternative Hypothesis: H a :  0 (2-sided alternative includes both > and <) Test Statistic: P-value: P=2P(Z  |z obs |)

6. Example - Mercury Levels Population: Patients visiting private internal medicine clinic in S.F. (High-end fish consumers) Variable: Mercury levels (microg/L) Sample: 66 Females Recommended maximum level: 5.0 microg/L Null hypothesis: H 0 :  = 5.0 (Mean level=RML) Alternative hypothesis: H a :   5.0 (Mean  RML) Sample Data:

7. Example - Mercury Levels Test Statistic: P-Value: P=2P(Z  5.41) < 2P(Z  5.00) = 2(.000000287)=.000000574  0 Conclusion: Very strong evidence that the population mean mercury level is above RML Source: Hightower and Moore (2003), “Mercury Levels in High-End Consumers of Fish, Environ Health Perspect, 111(4):A233

8. Miscellaneous Comments Effect of sample size on P-values: For a given observed sample mean and standard deviation, the larger the sample size, the larger the test statistic and smaller the P-value (as long as the sample mean does not equal  0 ) Equivalence between 2-tailed tests and confidence intervals: If a (1-  )100% CI for  contains  0, the P-value will be larger than  1-sided tests: Sometimes researchers have a specific direction in mind for alternative hypothesis prior to collecting data.

9. Example - Crime Rates (1960-80) Sample: n=74 Chicago Neighborhoods Goal: Show the average delinquency rate in the population of all such neighborhoods has increased from 1960-1980 Variable: Y = DR 1980 -DR 1960 H 0 :  = 0 (No change from 1960-1980) H a :  > 0 (Higher in 1980, see Y above) Sample Data:

10. Example - Crime Rates (1960-80) Test Statistic: P-value: (Only interested in larger positive values since 1-sided) Conclusion: Strong evidence that the true mean delinquency rate among all neighborhoods that this sample was taken from has increased from 1960 to 1980. Source: Bursik and Grasmick (1993), “Economic Deprivation and Neighborhood Crime Rates, 1960- 1980”, Law & Society Review, Vol. 27, pp 263-284

11. Significance Test for a Proportion (Large-Sample) Assumptions: –Qualitative Variable –Random sample –Large sample: n  10/min(  0, 1-  0 ) Hypotheses: –Null hypothesis: H 0 :  0 –Alternative hypothesis: H a :  0 (2-sided) –H a + :  0 H a - :  0 (1-sided, prior to data)

12. Significance Test for a Proportion (Large-Sample) Test statistic: P-value: –H a :  0 P = 2P(Z  |z obs |) –H a + :  0 P = P(Z  z obs ) –H a - :  0 P = P(Z  z obs ) Conclusion: Similar to test for a mean

13. Decisions in Tests  -level (aka significance level): Pre-specified “hurdle” for which one rejects H 0 if the P-value falls below it. (Typically.05 or.01) Rejection Region: Values of the test statistic for which we reject the null hypothesis For 2-sided tests with  =.05, we reject H 0 if |z obs |  1.96

14. Error Types Type I Error: Reject H 0 when it is true Type II Error: Do not reject H 0 when it is false

15. Error Types Probability of a Type I Error:  -Level (significance level) Probability of a Type II Error:  - depends on the true level of the parameter (in the range of values under H a ). For a given sample size, and variability in data, the Type I and Type II error rates are inversely related Conclusions wrt H 0 are the same whether a hypothesis test or CI is conducted (fixed  )

16. Miscellaneous Issues Statistical vs Practical Significance: With very large sample sizes, we can often obtain very small P-values even when the sample quantity is very close to the parameter value under H 0. Always consider the estimate as well as P-value. While hypothesis tests and confidence intervals give similar conclusions wrt H 0, the CI gives a credible set of parameter values, which can be more specific than test

17. Small-sample Inference for  t Distribution: –Population distribution for a variable is normal –Mean , Standard Deviation  –The t statistic has a sampling distribution that is called the t distribution with (n-1) degrees of freedom: Symmetric, bell-shaped around 0 (like standard normal, z distribution) Indexed by “degrees of freedom”, as they increase the distribution approaches z Have heavier tails (more probability beyond same values) as z Table B gives t A where P(t > t A ) = A for degrees of freedom 1-29 and various A

18. Small-Sample 95% CI for  Random sample from a normal population distribution: t.025,n-1 is the critical value leaving an upper tail area of.025 in the t distribution with n-1 degrees of freedom For n  30, use z.025 = 1.96 as an approximation for t.025,n-1

19. t test for a mean Assumptions: Random sample for a quantitative variable with a normal probability distribution Hypotheses: –H 0 :    a :  0 (2-sided) Test Statistic: P-Value: 2P(t > |t obs |) Conclusions as before, as well as 1-sided tests