1. Part IV – When the MOE Doesn’t Apply
Statistical Sampling 1

2. This video is designed to accompany pages 41-76 in
Making Sense of Uncertainty
Activities for Teaching Statistical Reasoning
Van-Griner Publishing Company 2

3. Non-Sampling Errors
The Margin of Error is a nice mathematical way of addressing sampling variability, also called “random sampling error.”
There are lots of other “errors” that can affect data collection and the MOE simply doesn’t apply. 3

4. Working Definition
Non-Sampling Error – An error or discrepancy caused by something other than the fact that a sample was selected instead of the entire population. These errors include, but are not limited to, data entry errors, nonresponse, biased questions in a questionnaire, question order, and false information provided by respondents. 4

5. Muddled Questions
“Do you plan to leave your job and look for another one during the coming year?”
“Have you often, sometimes, hardly ever, or never felt bad because you were unfaithful to your wife?”
(Harris Poll: One percent said often, 14 percent said sometimes or hardly ever, and 85 percent said they never felt bad because of this.)
“Should we increase taxes in order to get more housing and better schools, or should we keep them about the same?” 5

6. Percentage Nonresponse
Sensitive Questions
Roger Tourangeau and Ting Yan, Psychological Bulletin, 2007, Vol. 133, No. 5, 859–883
“Since 1971, the federal government has sponsored a series of recurring studies to estimate the prevalence of illicit drug use. Other surveys ask national samples of women whether they have ever had an abortion or ask samples of adults whether they voted in the most recent election. “
“An important question about such surveys is whether respondents answer the questions truthfully. Methodological research on the accuracy of reports in surveys about illicit drug use and other sensitive topics … is a major source of error, more specifically of bias, in the estimates derived from these surveys.”
Item
Percentage Nonresponse
Total household income
8.15
No. of lifetime male sexual partners
3.05
Received public assistance
2.22
No. of times had sex in past 4 weeks
1.37
Age of first sexual intercourse
0.87
Blood tested for HIV
0.65
Age of first menstrual period
0.39
Highest grade completed
0.04 6

7. How bad is it?
No hard data to compare sampling errors to non-sampling errors, but it is generally thought that non-sampling errors might be the larger of the two.
Harris Poll Disclaimer:
All sample surveys and polls, whether or not they use probability sampling, are subject to multiple sources of error which are most often not possible to quantify or estimate, including sampling error, coverage error, error associated with nonresponse, error associated with question wording and response options, and post-survey weighting and adjustments. Therefore, Harris Interactive avoids the words “margin of error” as they are misleading. 7

8. What to do?
Large body of literature on things to try and things that seem to work
Use of inducements for non-responders
Use of technology-assisted confidential interview techniques
Awareness of psychology of question order
Awareness of experiences with question wording
Use of internal consistency checks 8

9. One-Sentence Reflection
Non-sampling errors can substantially detract from the accuracy of survey data, and are not addressed by the margin of error.