FINAL REPORT: OUTLINE & OVERVIEW OF SURVEY ERRORS Lu Ann Aday, Ph.D. The University of Texas School of Public Health

STEPS IN DESIGNING AND CONDUCTING HEALTH SURVEYS

TYPES OF SURVEY ERRORS Systematic Error (bias): Difference (positive or negative) between survey estimate and actual population value Variable (random) Error: Range or variation (variance) in values of an estimate across observations (or cases)

EFFECTS OF SURVEY ERRORS ON SURVEY ESTIMATES

OVERVIEW OF SURVEY ERRORS SURVEY PROCEDURE SYSTEMATIC ERROR VARIABLE ERROR Study Design Poor internal validity Poor external validity Design specification ambiguity Sample Design Noncoverage bias Weighting errors Standard errors Design effects Data Collection Unit nonresponse bias Item nonresponse bias Interviewer variability Mode effects Questionnaire Design Under/over-reporting Yea-saying Order & context effects Measurement Low or poor validity Low or poor reliability Data Preparation Imputation/Estimation errors Data coding, editing, or data entry errors Analysis Plan Poor statistical conclusion validity Low statistical precision or power

SURVEY ERRORS: Study Design Systematic Poor internal validity. The study design does not adequately and accurately address the study’s hypotheses, particularly with respect to demonstrating a causal relationship between the independent (predictor) and dependent (outcome) variables. Poor external validity. Findings based on the study design cannot be widely or universally applied to related populations or subgroups. Variable Design specification ambiguity. The statement of the study objectives and related concepts to be measured in the survey are not clearly and unambiguously stated, particularly in relationship to the underlying study design and data analysis plan for the study.

SURVEY ERRORS: Sample Design Systematic Noncoverage bias. All units of the target population (e.g., households, individuals) are not included in the sampling frame (frame bias) or the respondent is not selected randomly (respondent selection bias). Weighting errors. Respondents are disproportionately represented in the survey sample by failing to weight each of the cases by the disproportionate probability of their falling into the sample (sampling fraction). Variable Standard errors. The standard error measures random sampling variation in an estimate (e.g., mean or proportion) across all possible random samples of a certain size that could theoretically be drawn from the target population. Design effects. The design effect, computed as the ratio of the variance of a complex sample to that of a simple random sample, measures the increase in random sampling variation in an estimate due to the complex nature of a sample design.

SURVEY ERRORS: Data Collection Systematic Unit nonresponse bias. Selected units of the study sample (e.g., households, individuals) are not included in the final study due to respondent refusals or unavailability during the data collection process. Item nonresponse bias. Selected questions on the survey questionnaire are not answered due to respondent refusals or interviewer or respondent errors or omissions during the data collection process. Variable Interviewer variability. Survey interviewers or data collectors vary in how they ask or record answers to the survey questions. Mode effects. The responses to comparable questions by respondents vary across different data collection methods (e.g., personal interview, telephone interview, mail self-administered questionnaire, web survey, etc.). [Note: If these effects differ in a particular direction across mode, they become systematic errors.]

SURVEY ERRORS: Questionnaire Design Systematic Under/over-reporting. An estimate (e.g., mean or proportion) across samples differs in a particular (negative or negative) direction from the underlying actual (or true) population value for the estimate, i.e., is lower (underreporting) or higher (overreporting). Yea-saying. Respondents tend to agree rather than disagree with statements as a whole (acquiescent response set) or with what are perceived to be socially desirable responses (social desirability bias). Variable Order & context effects. Answers to selected survey questions vary depending on whether they are asked before or after other questions and/or appear at the beginning or the end of the survey questionnaire.

SURVEY ERRORS: Measurement Systematic Low or poor validity. Systematic departures exist in answers to the content of a survey question from the meaning of the concept itself (content validity), a criterion for what constitutes an accurate answer based on another data source (criterion validity), and/or hypothesized relationships of the concept being measured with other measures or concepts (construct validity). Variable Low or poor reliability. Random variation exists in answers to a survey question due to when it is asked (test-retest reliability), who asked it (inter-rater reliability), and/or that it is simply one of a number of questions that could have been asked to obtain the information (internal consistency reliability).

SURVEY ERRORS: Data Preparation Systematic Imputation/Estimation errors. Procedures for assigning values to survey questions for which answers are not available (missing values due to item nonresponse) either from data internal or external to the survey (imputation or estimation, respectively) introduce systematic errors (biases) in estimating or examining relationships between variables. Variable Data coding, editing, or data entry errors. Data coding, editing, or data entry personnel or procedures introduce random errors in producing data files based on the survey questionnaires.

SURVEY ERRORS: Analysis Plan Systematic Poor statistical conclusion validity. The accuracy of statistical conclusions is compromised due to the application of statistical procedures that do not meet underlying assumptions related to the study design and objectives, level of measurement of study variables, and/or the underlying population distribution. Variable Low statistical precision or power. There are insufficient cases in the study sample to estimate population parameters with a reasonable level of precision or to have enough statistical power to detect statistically (and substantively) significant relationships between variables if they do exist.