1. Sources of Error In Sampling
Chapter 1 Section 4
Sources of Error
In Sampling

2. Chapter 1 – Section 4 Learning objectives
Understand how error can be introduced during sampling 1

3. Chapter 1 – Section 4
We will have errors when we calculate from a sample
Some errors can be minimized
Poor design of the sampling frame
Poor design of the sample questions
Some errors are difficult to control in sampling
Incomplete information compared to the entire population

4. Chapter 1 – Section 4
One type of error, sampling errors, occur because we use only part of the population in our study
Samples consist of only part of the total data
Samples are usually more realistic to analyze
Because there are individuals in the population that are not in our sample, sampling errors are difficult to control
We will study sampling errors in future chapters
One type of error, sampling errors, occur because we use only part of the population in our study
Samples consist of only part of the total data
Samples are usually more realistic to analyze
Because there are individuals in the population that are not in our sample, sampling errors are difficult to control
One type of error, sampling errors, occur because we use only part of the population in our study
Samples consist of only part of the total data
One type of error, sampling errors, occur because we use only part of the population in our study
One type of error, sampling errors, occur because we use only part of the population in our study
Samples consist of only part of the total data
Samples are usually more realistic to analyze

5. Chapter 1 – Section 4
Another type of error, nonsampling errors, occur from the actual survey process
Preference is given to selecting some individuals over others
Individual answers are not accurate (for various reasons)
Nonsampling errors can often be controlled or minimized with a well-designed survey and sampling technique

6. Chapter 1 – Section 4 Types of nonsampling error
Using an incomplete frame
Individuals who respond have different characteristics than individuals who do not respond
Interviewer errors
Misrepresented answers
Data checks
Questionnaire design
Types of nonsampling error
Using an incomplete frame
Individuals who respond have different characteristics than individuals who do not respond
Interviewer errors
Misrepresented answers
Data checks
Questionnaire design
Wording of questions
Types of nonsampling error
Using an incomplete frame
Individuals who respond have different characteristics than individuals who do not respond
Interviewer errors
Misrepresented answers
Data checks
Questionnaire design
Wording of questions
Order of questions, words, and responses
Types of nonsampling error
Using an incomplete frame
Individuals who respond have different characteristics than individuals who do not respond
Interviewer errors
Misrepresented answers
Data checks
Types of nonsampling error
Using an incomplete frame
Individuals who respond have different characteristics than individuals who do not respond
Interviewer errors
Types of nonsampling error
Types of nonsampling error
Using an incomplete frame
Types of nonsampling error
Using an incomplete frame
Individuals who respond have different characteristics than individuals who do not respond
Types of nonsampling error
Using an incomplete frame
Individuals who respond have different characteristics than individuals who do not respond
Interviewer errors
Misrepresented answers

7. Chapter 1 – Section 4 Using an incomplete frame
If the sample is not representative of the population, then the descriptive statistics of the sample may be very different from the true population parameters
This is similar to performing a stratified sample but with the incorrect strata percentages

8. Chapter 1 – Section 4
The Literary Digest used their polls to predict the winner of presidential elections
Their previous polls were accurate
In 1936, the Literary Digest predicted that Alf Landon would defeat Franklin Roosevelt in a landslide
In the actual election, Roosevelt won in a landslide
The Literary Digest used their polls to predict the winner of presidential elections
Their previous polls were accurate
In 1936, the Literary Digest predicted that Alf Landon would defeat Franklin Roosevelt in a landslide
The Literary Digest used their polls to predict the winner of presidential elections
Their previous polls were accurate
The Literary Digest used their polls to predict the winner of presidential elections

9. Chapter 1 – Section 4 Why was the Literary Digest so far off?
The 1936 frame was not representative of the total voting population
The sampling process was not completely random
The frame had too large of a proportion of Republicans, who generally favored Landon
The frame had too small of a proportion of Democrats, who generally favored Roosevelt
Republicans were overrepresented and Democrats were underrepresented
Why was the Literary Digest so far off?
The 1936 frame was not representative of the total voting population
The sampling process was not completely random
The frame had too large of a proportion of Republicans, who generally favored Landon
The frame had too small of a proportion of Democrats, who generally favored Roosevelt
Why was the Literary Digest so far off?
The 1936 frame was not representative of the total voting population
The sampling process was not completely random
Why was the Literary Digest so far off?
The 1936 frame was not representative of the total voting population
Why was the Literary Digest so far off?
The 1936 frame was not representative of the total voting population
The sampling process was not completely random
The frame had too large of a proportion of Republicans, who generally favored Landon

10. Chapter 1 – Section 4
Some individuals selected in the sample may not respond
People who do not respond may feel differently than people who do respond
This can be controlled with callbacks and incentives
An extreme case of this error occurs in self-selected samples

11. Chapter 1 – Section 4 Interviewer errors may occur when
The interviewer has a vested interest in the results
The interviewer is not trained to obtain accurate information
The individuals feel pressure or an obligation to provide an answer that the interviewer desires
For example, if your server watches you when you fill out the restaurant’s service satisfaction questionnaire …
Interviewer errors may occur when
The interviewer has a vested interest in the results
The interviewer is not trained to obtain accurate information
The individuals feel pressure or an obligation to provide an answer that the interviewer desires
Interviewer errors may occur when
The interviewer has a vested interest in the results
The interviewer is not trained to obtain accurate information
Interviewer errors may occur when
The interviewer has a vested interest in the results

12. Chapter 1 – Section 4
Some individuals may deliberately lie in their responses
Even if the other aspects of the survey are designed well, the individual may want to “look good” or may be embarrassed about giving truthful answers

13. Chapter 1 – Section 4
Data recorded by the individuals may be entered incorrectly onto the questionnaire
Data recorded by hand can often be entered into the system incorrectly (“typos”)
There are some techniques for detecting these errors, such as examining outliers

14. Chapter 1 – Section 4
Sometimes the questionnaire design affects the response
An open ended question is one where the respondent is free to choose the response
A closed ended question is one where the respondent must choose one in a predetermined list of responses
Each has their advantages
Sometimes the questionnaire design affects the response
An open ended question is one where the respondent is free to choose the response
A closed ended question is one where the respondent must choose one in a predetermined list of responses
Sometimes the questionnaire design affects the response
An open ended question is one where the respondent is free to choose the response
Sometimes the questionnaire design affects the response

15. Chapter 1 – Section 4 Open ended questions Closed ended questions
Allows the respondent to choose their own answer
Gives the flexibility to represent a variety of options
Closed ended questions
Limits the number of possible responses, making the analysis easier
Gives the respondents a structure explaining more of the meaning and purpose of the question
A combination of open ended and closed ended questions could be effective
Open ended questions
Allows the respondent to choose their own answer
Gives the flexibility to represent a variety of options
Open ended questions
Allows the respondent to choose their own answer
Gives the flexibility to represent a variety of options
Closed ended questions
Limits the number of possible responses, making the analysis easier
Gives the respondents a structure explaining more of the meaning and purpose of the question

16. Chapter 1 – Section 4
Wording of questions could affect the accuracy of responses
Some questions are not asked in balanced form
Some questions are worded in confusing ways
Some questions are too vague
The same question, worded in two different confusing ways, could result in two different results
Wording of questions could affect the accuracy of responses
Some questions are not asked in balanced form
Some questions are worded in confusing ways
Some questions are too vague

17. Chapter 1 – Section 4 Two questions for students
Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?”
Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”

18. Chapter 1 – Section 4 Two questions for students
Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?”
Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”

19. Chapter 1 – Section 4 Two questions for students
Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?”
Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”

20. Chapter 1 – Section 4 Two questions for students
Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?”
Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”
What will students say?
May tend to say “yes” to both questions, even though they are complete opposites
May tend to say “yes” because both have the words “no” and “required” and “2½ hours”
Two questions for students
Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?”
Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”
What will students say?
Two questions for students
Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?”
Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”
What will students say?
May tend to say “yes” to both questions, even though they are complete opposites

21. Chapter 1 – Section 4
The order of questions, words, and responses can affect the responses
The responses to two questions may differ if they are ordered as Question A then Question B, as compared to if they are ordered as Question B then Question A
One example is when many unfamiliar candidates are running for a position … the order of listing their names affects the number of votes they get
The order of names is often drawn at random
The order of questions, words, and responses can affect the responses
The responses to two questions may differ if they are ordered as Question A then Question B, as compared to if they are ordered as Question B then Question A

22. Summary: Chapter 1 – Section 4
There are reasons why samples have errors
Sampling errors are difficult to control because we use only part of the total population
Nonsampling errors are often largely avoidable with a good study design
Minimizing nonsampling errors is of high priority in designing a sample survey