Tests of Significance June 11, 2008 Ivan Katchanovski, Ph.D. POL 242Y-Y

Tests of Statistical Significance Tests of Statistical Significance: Formal and exact way to test hypotheses – Derived with help of advanced mathematics – Is a relationship between independent and dependent variables statistically significant? Widely used in the social sciences – Often misused Focus on application in research methods 2

Null Hypothesis Research Hypothesis (H 1 ) A statement about relationship between independent and dependent variables that we want to prove or disprove. – Example: people with college education have higher incomes than people with high school education Null Hypothesis (H 0 ) A statement of "no difference” between independent and dependent variables – Example: people with college education and people with high school education have the same incomes 3

Statistical Significance The null hypothesis: Dependent and independent variables are statistically unrelated If a relationship between an independent variable and a dependent variable is statistically nonsignificant – Null hypothesis is true – Research hypothesis is rejected If a relationship between an independent variable and a dependent variable is statistically significant – Null hypothesis is false – Research hypothesis is supported 4

Criteria of Statistical Significance Statistical significance: SPSS p(obtained)<p.=.001, or p=.01, or p=.05 Conventional levels of statistical significance: Less than.001: Probability that a tested relationship occurred by chance is less than.001, or 1 in 1000, or.1% Less than.01: Probability that a tested relationship occurred by chance is less than.01, or 1 in 100, or 1% Less than.05: Probability that a tested relationship occurred by chance is less than.05, or 1 in 20, or 5% Less than.10 (can be used if N is small) 5

6 Chi Square Test of Significance ( ) Can be used with variables at any level of measurement – Most appropriate for nominal and ordinal variables – Used in cross-tabulation analysis Pearson’s Chi square distribution – Karl Pearson – Eugenics Limitations – Problematic if expected frequencies in cells are small (5 or less)

7 Chi Square Distribution

8 Steps of Hypothesis Testing using Chi Square Step 0. Research hypothesis – Example: political party support in the US differs by gender Step 1. Assumptions: independent random sampling; variables are at nominal level of measurement Step 2. Null Hypothesis: The dependent and the independent variables are not related – Example: political party support is not related to gender Step 3. Selecting sampling distribution: SPSS does this automatically – Example: Chi-Square

9 Steps of Hypothesis Testing using Chi Square (Cont.) Step 4. Computing the test statistic using Chi-square formulas or SPSS command (Crosstabs) Step 5. Making a decision whether to reject or accept the null hypothesis. – If test statistic falls in the critical region: – SPSS p(obtained)<p=.05 Reject the null hypothesis and accept research hypothesis – Statistically insignificant if test statistic (Chi-Square) does not fall in the critical region: – SPSS p(obtained)>p=.05 Accept the null hypothesis and reject research hypothesis

10 Example: Political Party Support by Gender Bivariate (two variables) table of frequency distribution The dependent variable (political party support) is in rows The independent variable (gender) is in columns Political partyMale, %Female, % Republican5037 Democrat5063 Total, %100 N Source: 1996 Lipset/Meltz Survey

Example: Chi Square Test SPSS Chi square test: – Pearson Chi Square value= – P = Pearson Chi Square value (16.219) falls in the critical region of Chi Square distribution (Determined manually) SPSS automatic determination of statistical significance – SPSS p(obtained)=0.000<p=.05: Statistically significant – Select the lowest level of statistical significance SPSS p(obtained)=0.000<p=.001 Reject the null hypothesis Accept the research hypothesis: – Political party support in the US differs by gender. – The difference is statistically significant at the.001 level 11

Limitations of Tests of Statistical Significance Type I error (alpha) - rejecting a true null hypothesis Type II (beta) - failing to reject a false null hypothesis Equating statistical significance with real-life significance – Computers made statistical tests easy and fast – Almost any relationship can become statistically significant in surveys with very large number of respondents – Statistical significance does not always means real-life significance 12