Quality indicators for measuring and enhancing the composition of survey response Q2008 – Special topic session, July 9 Jelke Bethlehem and Barry Schouten

Session programme Enhancing response by differentiated data collection strategies Barry Schouten and Jelke Bethlehem Management tools for enhancing the composition of survey response Anne Sundvoll and Øyvin Kleven Use of process data to determine the number of call attempts in a telephone survey Annica Isaksson, Peter Lundquist and Daniel Thorburn An Indicator of Nonresponse Bias Derived from Call-back Analysis Paul Biemer Discussion

Enhancing response by differentiated data collection strategies Q2008 – Quality indicators for measuring and enhancing the composition of survey response Barry Schouten and Jelke Bethlehem

Enhancing the composition of response Session papers have in common that they want to minimize the effects of nonresponse by using auxiliary information and proces information (paradata) to either change the data collection or adjustment strategies. In this paper we propose to differentiate data collection strategies as a function of auxiliary information and paradata.

Differentiated data collection? Responsive designs - Groves & Heeringa (2006): The ability to monitor continually the streams of process data and survey data creates the opportunity to alter the design during the course of data collection to improve survey cost efficiency and to achieve more precise, less biased estimates. Responsive design: Learning period during fieldwork (phases) Especially useful for new or infrequent surveys with little available auxiliary information Data collection differentiated at macro level

Differentiated data collection? What if a survey is running for a long time and auxiliary information is available beforehand? Differentiate data collection at micro level: Different strategies for different households/persons/businesses Static: Independent of paradata Dynamic: Depending on paradata Goal: Optimal balance between quality and costs

What are ingredients to differentiated strategies? We need: 1. Auxiliary information 2. Strategies 3. Optimization criteria 4. Tools Model

Auxiliary information Auxiliary information: Must be available beforehand for full sample, or Can be collected during fieldwork for full sample. Sources: Frame data Registers and administrative data Paradata (fieldwork staff, interviewers) Crucial: Need for models and auxiliary variables that give a strong and consistent explanation of the different types of nonresponse (non-contact, refusal).

Data collection strategies Strategy is a series of pre-defined decision rules that depend only on auxiliary variables from frame and registers, and possibly also on paradata. Components: Data collection modes (f2f, telephone, web) Incentives Advance letters Reminders Contact strategy (Isaksson, Lundquist, Thorburn) Refusal conversion or call-back (Biemer)

Modelling strategies and response X Y S C,P R I Sample unit Paradata

Notation Strategies: Auxiliary: Distribution: Costs: Response: Allocation: with

Sample size, costs and response rate Expected sample size Expected costs Expected response rate

Objective function Criterion: conditional on Obvious choice: MSE conditional on

Objective function; example 7th Frame work Programme project RISQ Proposal: Minimize impact under worst case scenario Maximal absolute bias Objective function: Estimated response probabilities

Strategy allocation and sampling designs Strong resemblance with traditional sample designs if we view strategy allocation probabilities and response probabilities as inclusion probabilities. Response is second stage in “design” Stratification of strategy allocation From a variance point of view it is optimal to make the product of response and strategy allocation probabilities proportional to target variable(s) in survey

Tools We need tools to: Allocate sample units to different strategies Translate paradata and auxiliary information to indicators that enable monitoring and controlling Follow strategy during fieldwork These tools must be compatible with existing survey management tools and must be user-friendly to fieldwork staff One of main objectives of FP7 project RISQ (Representativity Indicators for Survey Quality)

Discussion Given a set of candidate strategies Can we predict response probabilities without bias? Is response behaviour sufficiently stable over time? What objective function to choose to optimize quality and costs? Is it realistic to employ individual cost functions? What demands come from fieldwork management?