Inferential Statistics (Part IV) Chapter 8 Significance Using Inferential Statistics A. New terminology in the research hypothesis: “There is a significant difference in _______ between groups; or a significant relationship between two variables.” B. If there is, then we assume that all other factors that might account for differences are “controlled” or considered. What level of risk are we willing to take in order to conclude that significant differences are present? This level is called the significance level. Example: Null Hypothesis: No difference in math scores between children who did and did not participate in summer school remedial math.

Purpose of Research: Show that observed differences are due ONLY to the program and not other factors. By concluding that the differences in test scores are due to differences in program exposure, we accept some risk. The level of risk accepted (risk of being wrong by rejecting null) is the significance level. Say our significance level is .05. We are saying that there is only 1 chance in 20 (or 5 in 100) that any differences found were not due to the hypothesized reason (program) but to some other factor.

Errors (Type I and Type II) A. Type I error: the null is true and you reject it. The significance level refers to how much risk we are willing to accept. If we set it at .05, then we are saying that there is a 5% chance that we are making a Type I error. The greater the standard the greater the chance. Easily controlled, just change the risk level. B. Type II: The null is false and you accept it. More subjects/cases, the closer to the true population, the lower the risk of making Type II errors. Not easily controlled, change the sample size.

Significance verses meaningfulness Just because a factor is significant, does not mean that it is meaningful. Sometimes, significant differences are negligible even if significant. Significance does not tell us how big the impact is, it only tells us if there is an impact. The more significant DOES NOT mean the larger the impact or strength. Example: Are you willing to retain children in Grade 1 if the retention program significantly raises their standardized test scores by one-half point?

Selecting a test (p. 166 Figure 9.1; Fall in Love With this Diagram) The test you select depends upon the data that you have and the question you are asking. Steps of Significance Testing: State the null: (ex: no difference between average number of men and women voting) Set the significance level (.05): There is only a 5% chance that observed differences are due to chance (95% due to something else, like gender). Select appropriate test statistic (figure 8.1): difference in means (next few chapters) Compute test statistic or obtained value (use SPSS and formulas). Determination of value needed for rejection of null using the appropriate table of critical values (corresponds to a significance level, usually 5%)

Compare obtained value to critical value If obtained value is more extreme than critical (less than 5% of all values), reject null. If not, do not reject null (chance is most reasonable explanation). That is: if ov>cv, reject null. If ov<cv, do not reject null. See Fig 9.2.