Statistical Significance What is Statistical Significance? What is Statistical Significance? How Do We Know Whether a Result is Statistically Significant? How Do We Know Whether a Result is Statistically Significant? Significance as a Probability Game Significance as a Probability Game What is a Sampling Distribution? What is a Sampling Distribution?

What is Statistical Significance? Statistical significance is a technical decision made using inferential statistics. Statistical significance is a technical decision made using inferential statistics. We say that a result is statistically significant if our inferential statistic indicates that we reject the Null Hypothesis. We say that a result is statistically significant if our inferential statistic indicates that we reject the Null Hypothesis.

How Do We Know Whether a Result is Statistically Significant? Test the Null Hypothesis using an inferential statistic. Test the Null Hypothesis using an inferential statistic. The result of the statistical test indicates a probability. If the probability is lower than our criterion significance level, we reject the Null, meaning that the result is significant.

Determining Significance The Null Hypothesis (H o ) states that there is no difference, effect, or correlation in the population The Null Hypothesis (H o ) states that there is no difference, effect, or correlation in the population H o is assumed to be true unless there is enough evidence to reject it. H o is assumed to be true unless there is enough evidence to reject it. Burden of proof on the researcher Burden of proof on the researcher The researcher’s hypothesis (Alternative Hypothesis, H A ) is only tested indirectly

Determining Significance How strong does the evidence have to be to reject the Null? How strong does the evidence have to be to reject the Null? The researcher must set a criterion. This is the significance level, or alpha (  ). The researcher must set a criterion. This is the significance level, or alpha (  ). The conventional alpha level is.05. The conventional alpha level is.05. We are conservative about rejecting Ho. We are conservative about rejecting Ho.

Determining Significance When testing for significance, we calculate a test statistic. When testing for significance, we calculate a test statistic. The test statistic allows us to determine the probability of obtaining our results under the assumption that H o is true. The test statistic allows us to determine the probability of obtaining our results under the assumption that H o is true. If this probability is small enough, then H o is probably not true, so we should reject it.

Determining Significance If the probability is lower than our significance level, we Reject Ho (p <.05). If the probability is lower than our significance level, we Reject Ho (p <.05). If the probability is not lower than our significance level, we Fail to Reject Ho (p >.05). If the probability is not lower than our significance level, we Fail to Reject Ho (p >.05). Ho is never “accepted” or “proven.” Ho is never “accepted” or “proven.”

Significance as a Probability Game There are four possible outcomes in significance test, based on two dimensions: There are four possible outcomes in significance test, based on two dimensions: The researcher’s decision about Ho. The researcher’s decision about Ho. Whether Ho is really true or false. Whether Ho is really true or false. The probability of each outcome can be determined. The probability of each outcome can be determined.

Ho true Ho false TRUE STATE OF THE WORLD DECISION RejectHo Fail to Reject Ho Type I error  Correct 1 -  Correct 1 -  (power) Type II error  (beta)

Statistics as a Probability Game  is set by the researcher  is set by the researcher 1-  depends on  1-  depends on 

Statistics as a Probability Game Power is increased by: Power is increased by: higher alpha higher alpha larger sample larger sample lower variability lower variability larger effect size larger effect size Anything that increases power decreases beta

What is a Sampling Distribution? A hypothetical frequency distribution of sample statistics from an infinite number of samples. Allows us to make probability judgments about the likelihood of obtaining a particular result.

Imagining a Sampling Distribution 1.Take a random sample. 2.Compute the mean. 3.Take another random sample and compute the mean. 4.Do this an infinite number of times. 5.Put the resulting sample means in a frequency distribution.

Nice Things About Sampling Distributions 1. The mean is the hypothesized population mean. 1. The mean is the hypothesized population mean. 2. The standard deviation can be calculated (standard error). 2. The standard deviation can be calculated (standard error). 3. The shape is usually normal. 3. The shape is usually normal.

Central Limits Theorem The sampling distribution becomes more normal as the sample size increases. With a sample size of 30 or more, the sampling distribution becomes very close to exactly normal.

Why These Are “Nice” Things If you know the  and  of a distribution, you can compute z-scores. If you know the  and  of a distribution, you can compute z-scores. In a normal distribution, you can look up the proportion of scores above or below any z score. In a normal distribution, you can look up the proportion of scores above or below any z score. For any sample mean in the sampling distribution, we can find the proportion of sample means above or below it. For any sample mean in the sampling distribution, we can find the proportion of sample means above or below it.

Making Inferences There are three distributions used when we make an inference: There are three distributions used when we make an inference: sample distribution sample distribution sampling distribution sampling distribution population distribution population distribution The sampling distribution is the “bridge” from the sample to the population. The sampling distribution is the “bridge” from the sample to the population.