1. FINAL REPORT: OUTLINE & OVERVIEW OF SURVEY ERRORS
Lu Ann Aday, Ph.D.
The University of Texas
School of Public Health

3. STEPS IN DESIGNING AND CONDUCTING HEALTH SURVEYS

4. TYPES OF SURVEY ERRORS
Systematic Error (bias): Difference (positive or negative) between survey estimate and actual population value
Variable (random) Error: Range or variation (variance) in values of an estimate across observations (or cases)

5. EFFECTS OF SURVEY ERRORS ON SURVEY ESTIMATES

6. OVERVIEW OF SURVEY ERRORS
SURVEY PROCEDURE
SYSTEMATIC ERROR
VARIABLE ERROR
Study Design
Poor internal validity
Poor external validity
Design specification ambiguity
Sample Design
Noncoverage bias
Weighting errors
Standard errors
Design effects
Data Collection
Unit nonresponse bias
Item nonresponse bias
Interviewer variability
Mode effects
Questionnaire Design
Under/over-reporting
Yea-saying
Order & context effects
Measurement
Low or poor validity
Low or poor reliability
Data Preparation
Imputation/Estimation errors
Data coding, editing, or data entry errors
Analysis Plan
Poor statistical conclusion validity
Low statistical precision or power

7. SURVEY ERRORS: Study Design
Systematic
Poor internal validity. The study design does not adequately and accurately address the study’s hypotheses, particularly with respect to demonstrating a causal relationship between the independent (predictor) and dependent (outcome) variables.
Poor external validity. Findings based on the study design cannot be widely or universally applied to related populations or subgroups.
Variable
Design specification ambiguity. The statement of the study objectives and related concepts to be measured in the survey are not clearly and unambiguously stated, particularly in relationship to the underlying study design and data analysis plan for the study.

8. SURVEY ERRORS: Sample Design
Systematic
Noncoverage bias. All units of the target population (e.g., households, individuals) are not included in the sampling frame (frame bias) or the respondent is not selected randomly (respondent selection bias).
Weighting errors. Respondents are disproportionately represented in the survey sample by failing to weight each of the cases by the disproportionate probability of their falling into the sample (sampling fraction).
Variable
Standard errors. The standard error measures random sampling variation in an estimate (e.g., mean or proportion) across all possible random samples of a certain size that could theoretically be drawn from the target population.
Design effects. The design effect, computed as the ratio of the variance of a complex sample to that of a simple random sample, measures the increase in random sampling variation in an estimate due to the complex nature of a sample design.

9. SURVEY ERRORS: Data Collection
Systematic
Unit nonresponse bias. Selected units of the study sample (e.g., households, individuals) are not included in the final study due to respondent refusals or unavailability during the data collection process.
Item nonresponse bias. Selected questions on the survey questionnaire are not answered due to respondent refusals or interviewer or respondent errors or omissions during the data collection process.
Variable
Interviewer variability. Survey interviewers or data collectors vary in how they ask or record answers to the survey questions.
Mode effects. The responses to comparable questions by respondents vary across different data collection methods (e.g., personal interview, telephone interview, mail self-administered questionnaire, web survey, etc.). [Note: If these effects differ in a particular direction across mode, they become systematic errors.]

10. SURVEY ERRORS: Questionnaire Design
Systematic
Under/over-reporting. An estimate (e.g., mean or proportion) across samples differs in a particular (negative or negative) direction from the underlying actual (or true) population value for the estimate, i.e., is lower (underreporting) or higher (overreporting).
Yea-saying. Respondents tend to agree rather than disagree with statements as a whole (acquiescent response set) or with what are perceived to be socially desirable responses (social desirability bias).
Variable
Order & context effects. Answers to selected survey questions vary depending on whether they are asked before or after other questions and/or appear at the beginning or the end of the survey questionnaire.

11. SURVEY ERRORS: Measurement
Systematic
Low or poor validity. Systematic departures exist in answers to the content of a survey question from the meaning of the concept itself (content validity), a criterion for what constitutes an accurate answer based on another data source (criterion validity), and/or hypothesized relationships of the concept being measured with other measures or concepts (construct validity).
Variable
Low or poor reliability. Random variation exists in answers to a survey question due to when it is asked (test-retest reliability), who asked it (inter-rater reliability), and/or that it is simply one of a number of questions that could have been asked to obtain the information (internal consistency reliability).

12. SURVEY ERRORS: Data Preparation
Systematic
Imputation/Estimation errors. Procedures for assigning values to survey questions for which answers are not available (missing values due to item nonresponse) either from data internal or external to the survey (imputation or estimation, respectively) introduce systematic errors (biases) in estimating or examining relationships between variables.
Variable
Data coding, editing, or data entry errors. Data coding, editing, or data entry personnel or procedures introduce random errors in producing data files based on the survey questionnaires.

13. SURVEY ERRORS: Analysis Plan
Systematic
Poor statistical conclusion validity. The accuracy of statistical conclusions is compromised due to the application of statistical procedures that do not meet underlying assumptions related to the study design and objectives, level of measurement of study variables, and/or the underlying population distribution.
Variable
Low statistical precision or power. There are insufficient cases in the study sample to estimate population parameters with a reasonable level of precision or to have enough statistical power to detect statistically (and substantively) significant relationships between variables if they do exist.