1. Lecture: statistical significance.
POSC 202A: Lecture 9
Lecture: statistical significance.

2. Statistical Significance
The fundamental question of statistical significance:

3. Statistical Significance
The fundamental Question:
How likely is the result we observe to be the product of chance?
This question drives all of the tests we perform by allowing us to differentiate the systematic from the stochastic.

4. Statistical Significance
Confidence Interval-
An interval calculated from sample data that is guaranteed to capture the true population parameter.
It tells us how large of an interval we need to create in order to capture the true population value in some fixed percentage of the intervals we draw.

6. Statistical Significance
Think of it this way:
We can draw a sample in order to estimate some statistic. If we repeat over and over we start to create a sampling distribution.
To create a 95% CI we need to see how large of an interval around the statistic we need to create to satisfy the condition that in 95% of the samples drawn the interval we draw contains the true population parameter (value).
Page 386 has a nice graph of this.

7. Statistical Significance
Confidence Interval:
An example: a 95% confidence interval is the range needed to capture the true population value in 95% of the intervals we draw from a population.

8. Statistical Significance
Confidence Interval:
The interval thus captures the variability inherent in using samples to draw inferences about a population.
To estimate, we need an estimate of the population mean and the standard deviation.
Reported in the form:
Estimate Margin of Error
Generally, the interval is:
mean – (z*sd); mean+(z*sd)

9. Statistical Significance
Confidence Interval:
To estimate, we need an estimate of the population mean and the standard deviation.
How do we calculate this for a sampling distribution?
This is for proportions:
Mean=p

10. Refresher: Thumb’s Rule
Recall that with a normal distribution:
Apportionment of area about the mean is
+/- 1sd= 68%
+/- 2sd= 95%
+/- 3sd= 99.7%
So, a 95% CI corresponds to

11. Statistical Significance
Lets create an example using a sample from Dear Abby, in which 400 women responded of whom 60% would rather just cuddle than have sex with their husbands.
What do we want to know?

12. Statistical Significance
What do we want to know?
Is 60% beyond what we would expect to see due to chance alone?
Is this likely to be just random variation?

13. Statistical Significance
Lets create an example using a sample from Dear Abby, in which 400 women responded of whom 60% would rather just cuddle than have sex with their husbands.
Sample mean=p
.6 =p, so: S.D.=

14. Our confidence interval is thus:
+/- 2sd= 95 CI Or
.6 +/- 2(.0245)= and

15. Confidence Interval .6 +/- 2(.0245)= .6-.049 and .6+.049
Round off to .05
95% CI
.55
.60
.65

16. Confidence Interval
95% of all samples will capture the true population parameter in the range between
95% CI
.55
.60
.65

17. Confidence Interval
95% of all samples will capture the true population parameter in the range between
From this we conclude that we are 95% confident that between 55% and 65% of women prefer cuddling to sex.

18. Statistical Significance
How do our estimates change with the size of the population?
Recall we found
The population average (mean) stays the same regardless of sample size. But what of the SD?

19. Sample Size and Confidence Intervals
Where the sample statistic is .5

21. A shortcut for approximating the 95% CI for a proportion:
Confidence Interval
A shortcut for approximating the 95% CI for a proportion:
More accurate as you get closer to an even split (i.e. 50%)

22. Confidence Interval .

23. Statistical Significance
Recall--The fundamental Question:
How likely is the result we observe to be the product of chance?
This question drives all of the tests we perform by allowing us to differentiate the systematic from the stochastic.
Significance testing is all about comparisons.

24. Statistical Significance
Significance testing is all about comparisons.
Is what we observe close or far from what we expect ?
Is what we observe so far from what we expect that we cannot attribute what we see to chance alone.

25. Statistical Significance
An example: Imagine we took a valid random sample of women’s preferences and got the same result as Dear Abby’s survey of women’s cuddling preferences (60% preferred cuddling to sex; 400 women responded).
How would we conduct a significance test?
We need to identify the appropriate comparison.

26. Statistical Significance
How would we conduct a significance test?
What if women randomly answered “snuggle” or “sex”?

27. Statistical Significance
What we would expect if women just answered randomly?
If so, we might expect 50% of respondents to prefer snuggling.

28. Statistical Significance
What we would expect if women just answered randomly?
If so, we might expect 50% of respondents to prefer snuggling.
Then we need to know if what we observed is (60%) too large of a result to be attributable to chance alone.
How can we determine this?

29. Statistical Significance
How can we determine this?
One way is to estimate a 95% confidence interval around our sample mean (60%) and see if it contains the result we would expect sue to chance alone (50%).

30. Confidence Interval
Recall, we created the following interval earlier. We can simply look to see if .50 is within the interval constructed.
Since it is not, then our sample result is statistically significantly different than chance.
95% CI
.50
.55
.60
.65

31. Statistical Significance
How can we determine this?
A second way is to conduct a significance test by solving for areas under the normal curve. Recall we know how to find the likelihood that some event occurs using the formula: .
This formula asks whether what we see is too far from chance to be attributed to chance alone.

32. Statistical Significance
We simply calculate the number o standard units what we observe is from random chance.
Then use the Z table to obtain the likelihood that we see a sample as large as 60% if the true value is 50%.

33. Statistical Significance
Our Z table only goes to 3.4!
Less than 1 time in 10,000 would we see a sample mean of 60% if the process were driven by chance alone.

34. Confidence Interval
We could illustrate this process on the normal curve as well.
The Q: How likely is it that we would see a result above 60%?
Z>.9998
Z<.0001
.50
.55
.60