Sources of Error In Sampling Chapter 1 Section 4 Sources of Error In Sampling

Chapter 1 – Section 4 Learning objectives Understand how error can be introduced during sampling 1

Chapter 1 – Section 4 We will have errors when we calculate from a sample Some errors can be minimized Poor design of the sampling frame Poor design of the sample questions Some errors are difficult to control in sampling Incomplete information compared to the entire population

Chapter 1 – Section 4 One type of error, sampling errors, occur because we use only part of the population in our study Samples consist of only part of the total data Samples are usually more realistic to analyze Because there are individuals in the population that are not in our sample, sampling errors are difficult to control We will study sampling errors in future chapters One type of error, sampling errors, occur because we use only part of the population in our study Samples consist of only part of the total data Samples are usually more realistic to analyze Because there are individuals in the population that are not in our sample, sampling errors are difficult to control One type of error, sampling errors, occur because we use only part of the population in our study Samples consist of only part of the total data One type of error, sampling errors, occur because we use only part of the population in our study One type of error, sampling errors, occur because we use only part of the population in our study Samples consist of only part of the total data Samples are usually more realistic to analyze

Chapter 1 – Section 4 Another type of error, nonsampling errors, occur from the actual survey process Preference is given to selecting some individuals over others Individual answers are not accurate (for various reasons) Nonsampling errors can often be controlled or minimized with a well-designed survey and sampling technique

Chapter 1 – Section 4 Types of nonsampling error Using an incomplete frame Individuals who respond have different characteristics than individuals who do not respond Interviewer errors Misrepresented answers Data checks Questionnaire design Types of nonsampling error Using an incomplete frame Individuals who respond have different characteristics than individuals who do not respond Interviewer errors Misrepresented answers Data checks Questionnaire design Wording of questions Types of nonsampling error Using an incomplete frame Individuals who respond have different characteristics than individuals who do not respond Interviewer errors Misrepresented answers Data checks Questionnaire design Wording of questions Order of questions, words, and responses Types of nonsampling error Using an incomplete frame Individuals who respond have different characteristics than individuals who do not respond Interviewer errors Misrepresented answers Data checks Types of nonsampling error Using an incomplete frame Individuals who respond have different characteristics than individuals who do not respond Interviewer errors Types of nonsampling error Types of nonsampling error Using an incomplete frame Types of nonsampling error Using an incomplete frame Individuals who respond have different characteristics than individuals who do not respond Types of nonsampling error Using an incomplete frame Individuals who respond have different characteristics than individuals who do not respond Interviewer errors Misrepresented answers

Chapter 1 – Section 4 Using an incomplete frame If the sample is not representative of the population, then the descriptive statistics of the sample may be very different from the true population parameters This is similar to performing a stratified sample but with the incorrect strata percentages

Chapter 1 – Section 4 The Literary Digest used their polls to predict the winner of presidential elections Their previous polls were accurate In 1936, the Literary Digest predicted that Alf Landon would defeat Franklin Roosevelt in a landslide In the actual election, Roosevelt won in a landslide The Literary Digest used their polls to predict the winner of presidential elections Their previous polls were accurate In 1936, the Literary Digest predicted that Alf Landon would defeat Franklin Roosevelt in a landslide The Literary Digest used their polls to predict the winner of presidential elections Their previous polls were accurate The Literary Digest used their polls to predict the winner of presidential elections

Chapter 1 – Section 4 Why was the Literary Digest so far off? The 1936 frame was not representative of the total voting population The sampling process was not completely random The frame had too large of a proportion of Republicans, who generally favored Landon The frame had too small of a proportion of Democrats, who generally favored Roosevelt Republicans were overrepresented and Democrats were underrepresented Why was the Literary Digest so far off? The 1936 frame was not representative of the total voting population The sampling process was not completely random The frame had too large of a proportion of Republicans, who generally favored Landon The frame had too small of a proportion of Democrats, who generally favored Roosevelt Why was the Literary Digest so far off? The 1936 frame was not representative of the total voting population The sampling process was not completely random Why was the Literary Digest so far off? The 1936 frame was not representative of the total voting population Why was the Literary Digest so far off? The 1936 frame was not representative of the total voting population The sampling process was not completely random The frame had too large of a proportion of Republicans, who generally favored Landon

Chapter 1 – Section 4 Some individuals selected in the sample may not respond People who do not respond may feel differently than people who do respond This can be controlled with callbacks and incentives An extreme case of this error occurs in self-selected samples

Chapter 1 – Section 4 Interviewer errors may occur when The interviewer has a vested interest in the results The interviewer is not trained to obtain accurate information The individuals feel pressure or an obligation to provide an answer that the interviewer desires For example, if your server watches you when you fill out the restaurant’s service satisfaction questionnaire … Interviewer errors may occur when The interviewer has a vested interest in the results The interviewer is not trained to obtain accurate information The individuals feel pressure or an obligation to provide an answer that the interviewer desires Interviewer errors may occur when The interviewer has a vested interest in the results The interviewer is not trained to obtain accurate information Interviewer errors may occur when The interviewer has a vested interest in the results

Chapter 1 – Section 4 Some individuals may deliberately lie in their responses Even if the other aspects of the survey are designed well, the individual may want to “look good” or may be embarrassed about giving truthful answers

Chapter 1 – Section 4 Data recorded by the individuals may be entered incorrectly onto the questionnaire Data recorded by hand can often be entered into the system incorrectly (“typos”) There are some techniques for detecting these errors, such as examining outliers

Chapter 1 – Section 4 Sometimes the questionnaire design affects the response An open ended question is one where the respondent is free to choose the response A closed ended question is one where the respondent must choose one in a predetermined list of responses Each has their advantages Sometimes the questionnaire design affects the response An open ended question is one where the respondent is free to choose the response A closed ended question is one where the respondent must choose one in a predetermined list of responses Sometimes the questionnaire design affects the response An open ended question is one where the respondent is free to choose the response Sometimes the questionnaire design affects the response

Chapter 1 – Section 4 Open ended questions Closed ended questions Allows the respondent to choose their own answer Gives the flexibility to represent a variety of options Closed ended questions Limits the number of possible responses, making the analysis easier Gives the respondents a structure explaining more of the meaning and purpose of the question A combination of open ended and closed ended questions could be effective Open ended questions Allows the respondent to choose their own answer Gives the flexibility to represent a variety of options Open ended questions Allows the respondent to choose their own answer Gives the flexibility to represent a variety of options Closed ended questions Limits the number of possible responses, making the analysis easier Gives the respondents a structure explaining more of the meaning and purpose of the question

Chapter 1 – Section 4 Wording of questions could affect the accuracy of responses Some questions are not asked in balanced form Some questions are worded in confusing ways Some questions are too vague The same question, worded in two different confusing ways, could result in two different results Wording of questions could affect the accuracy of responses Some questions are not asked in balanced form Some questions are worded in confusing ways Some questions are too vague

Chapter 1 – Section 4 Two questions for students Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”

Chapter 1 – Section 4 Two questions for students Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”

Chapter 1 – Section 4 Two questions for students Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”

Chapter 1 – Section 4 Two questions for students Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?” What will students say? May tend to say “yes” to both questions, even though they are complete opposites May tend to say “yes” because both have the words “no” and “required” and “2½ hours” Two questions for students Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?” What will students say? Two questions for students Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?” What will students say? May tend to say “yes” to both questions, even though they are complete opposites

Chapter 1 – Section 4 The order of questions, words, and responses can affect the responses The responses to two questions may differ if they are ordered as Question A then Question B, as compared to if they are ordered as Question B then Question A One example is when many unfamiliar candidates are running for a position … the order of listing their names affects the number of votes they get The order of names is often drawn at random The order of questions, words, and responses can affect the responses The responses to two questions may differ if they are ordered as Question A then Question B, as compared to if they are ordered as Question B then Question A

Summary: Chapter 1 – Section 4 There are reasons why samples have errors Sampling errors are difficult to control because we use only part of the total population Nonsampling errors are often largely avoidable with a good study design Minimizing nonsampling errors is of high priority in designing a sample survey