Nonresponse Rates and Nonresponse Bias In Surveys Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA Emilia Peytcheva University of Michigan, USA Funding from the Methodology, Measurement, and Statistics Program of the US National Science Foundation, Grant

Four Mutually-Problematic Observations 1.With 100% response rates probability sampling offers an inferential paradigm with measurable uncertainties for unbiased estimates 2.Response rates are declining 3.Keeter et al. (2000), Curtin et al. (2000), Merkle and Edelman (2002) show no nonresponse bias associated with varying nonresponse rates 4.Practitioners are urged to achieve high response rates Result: Confusion among practitioners

Assembly of Prior Studies of Nonresponse Bias Search of peer-reviewed and other publications 47 articles reporting 59 studies About 959 separate estimates (566 percentages) –mean nonresponse rate is 36% –mean bias is 8% of the full sample estimate We treat this as 959 observations, weighted by sample sizes, multiply-imputed for item missing data, standard errors reflecting clustering into 59 studies and imputation variance

Percentage Absolute Relative Bias

Percentage Absolute Relative Nonresponse Bias by Nonresponse Rate for 959 Estimates from 59 Studies

Conclusions from 959 Estimates Examples of large nonresponse bias exist Variation in nonresponse bias lies mostly among estimates within the same survey The nonresponse rate by itself is not a good predictor of nonresponse bias [Note: We cannot infer from the scatterplot about what would happen within a study if response rates were increased]

Thinking Causally About Nonresponse Rates and Nonresponse Error Key scientific question concerns mechanisms of response propensity that create covariance with survey variable where is the covariance between the survey variable, y, and the response propensity, p What mechanisms produce the covariance?

Alternative Causal Models for Studies of Nonresponse Rates and Nonresponse Bias

Types of Hypotheses about Influences on Nonresponse Error Influences on response rates –urbanicity, gender, age –topic of survey, population’s interest in topic –mode of data collection –prenotification, incentives Sponsorship –prior involvement of population with sponsor –government vs. other sponsor Types of measures –attitudinal vs. behavioral –questions related to topic of survey vs. others Type of statistic –means on counts/continuous variables vs. percentages –differences of subclass means

Types of Hypotheses about Influences on Nonresponse Error Influences on response rates –urbanicity, gender, age –topic of survey, population’s interest in topic –mode of data collection –prenotification, incentives Sponsorship –prior involvement of population with sponsor –government vs. other sponsor Types of measures –attitudinal vs. behavioral –questions related to topic of survey vs. others Type of statistic –mean vs. percentages –differences of subclass means Attributes of Surveys Attributes of Estimates

Exploratory Analysis in Two Steps Examine only estimates that are percentages, using standardized values of the percentages Step 1: pooling all 566 estimates, examine, difference of respondent and nonrespondent means Step 2: separating the estimates by their survey’s response rate, examine, nonresponse bias

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Type of Population diff=.075 ste=.033

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Type of Population diff=.075 diff= ste=.033 ste=

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Mode diff=.040 diff= ste=.024 ste=

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Involvement with Sponsor diff=.052 diff= ste=.022 ste=

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Type of Measure diff=.14 diff= ste=.022 ste=

Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Statistic’s Relevance to Topic diff=.006 diff= ste=.023 ste=

Respondent-Nonrespondent Functions on All Estimates by Type of Estimator diff=4.37 diff= ste=0.72 ste=

Do Differences of Subclass Means have Lower Nonresponse Bias? When estimating subclass differences, we hope that nonresponse biases of the two estimates cancel 120 reported estimates of subclass means and their differences Only 45 of them have bias of the differences of subclass means lower than average bias of the two subclass means –this comports with only 45 having two subclass means with biases of the same sign

Absolute Value of Bias of Difference of Subclass Mean by Absolute Value of Subclass Mean

Types of Hypotheses about Influences on Nonresponse Error Influences on response rates –urbanicity, gender, age –topic of survey, population’s interest in topic –mode of data collection –prenotification, incentives Sponsorship –prior involvement of population with sponsor –government vs. other sponsor Types of measures –attitudinal vs. behavioral –questions related to topic of survey vs. others Type of statistic –mean vs. percentages –differences of subclass means

Five Summary Statements 1.Large nonresponse biases exist 2.Most variation in nonresponse biases lie among estimates in the same survey 3.Some types of surveys are more susceptible to biases (e.g., interviewer-administered, studies of population without prior involvement with the sponsor, general population surveys) 4.Some types of estimates seem more susceptible to biases (e.g., measures of attitudes, percentages, (maybe) estimates related to survey topic) 5.Differences of subclass means do not tend to have lower biases that the individual subclass means Note: preliminary and exploratory analysis; there is much more to do