Mode effects in social surveys A mixed-mode experiment linked to the Safety Monitor Ger Linden, Leanne Houben, Barry Schouten (Statistics Netherlands)

Mode effects in social surveys – taskforce SASU, Feb 18 1 Summary The impact of the survey mode Do we expect mode effects in surveys on safety? How to deal with mode effects? How to measure mode effects? Relevance to SASU and ESS

Mode effects in social surveys – taskforce SASU, Feb 18 2 The impact of the survey mode Data collection steps: 1.Persons need to be reached (coverage) 2.Persons need to respond (response) 3.Persons need to provide valid answers (measurement) Representative

Mode effects in social surveys – taskforce SASU, Feb 18 3 The impact of the survey mode Coverage and response are usually measured in terms of coverage rate and response rate. Coverage and response form the “selection-effect”. Mode CoverageResponseTotal Face-to-face100%65% Telephone70%65%45% Web85%30%25% Web + paper100%35% Registered phone and web populations are known to be different from overall population.

Mode effects in social surveys – taskforce SASU, Feb 18 4 The impact of the survey mode Differences in measurement as a consequence of response styles per mode, “pure mode-effect” or “measurement effect”. Response styles identified through model of Tourangeau & Rasinski: Interpretation Information retrieval Judgement & processing Reporting Satisficing Social desirability

Mode effects in social surveys – taskforce SASU, Feb 18 5 The impact of the survey mode

Mode effects in social surveys – taskforce SASU, Feb 18 6 The impact of the survey mode – an example Items (categories 1 – 5) Estimate phone – estimate web Perception of economic development over past year Expected economic development for coming year Perception of change personal finances over past year Expected change personal finances for coming year Expected change in consumption: durable goods Pilot study Survey on Consumer Satisfaction – web and telephone

Mode effects in social surveys – taskforce SASU, Feb 18 7 Do we expect mode effects in safety surveys? Literature has shown that selection effects are mode- specific for general demographics and socio-economic background. Specific selection effects in safety surveys: Persons interested in the topic Persons afraid to let unknown persons in the house Persons that do not pick up unknown phone numbers Persons that experience feelings of unsafety

Mode effects in social surveys – taskforce SASU, Feb 18 8 Do we expect mode effects in safety surveys? Safety Monitor LFSHealth Survey Consumer Satisfaction Socially desirable answeringX(X)XX Memory effects (under-reporting)XXX- Memory effects (over-reporting)X(X)X- Memory effects (telescoping)X-X- Non-differentiationX--X Random ResponseX--X AcquiescenceX(X)-X Order effects (primacy, recency)X-XX No opinionX(X) X

Mode effects in social surveys – taskforce SASU, Feb 18 9 Do we expect mode effects in safety surveys? Consequences of mode effects: Impact on comparability of statistics over time Impact on comparability between publication domains For example: Young persons prefer self-assisted modes and these modes lead to more satisficing to batteries of questions on perception of safety or victimisation.

Mode effects in social surveys – taskforce SASU, Feb How to deal with mode effects? Adapt methodology: Data collection strategy and choice of modes Questionnaire design Estimation strategy General rule: avoid contrasts between modes Effectiveness of methodology depends on selection and pure mode effects, i.e. we need to disentangle the mode effects!

Mode effects in social surveys – taskforce SASU, Feb How to measure mode effects? Experimental design linked to Safety Monitor Sample = 8000 persons CAPICATICAWIPAPI CAPI + CATI Golf 1 Golf 2 ResponseNonresponse Principle: Selection effects and pure mode effects are measured relative to face-to-face interviews

Mode effects in social surveys – taskforce SASU, Feb How to measure mode effects? Wave 1: Random assignment to survey mode Questionnaire is Safety Monitor with small modifications Regular data collection strategy

Mode effects in social surveys – taskforce SASU, Feb How to measure mode effects? Wave 2: Almost full sample is observed face-to-face Questionnaire repeats main wave 1 questions Additional questions about general survey attitudes, attitudes towards politics and survey design features. Wave 1 mode effects are measured and disentangled by weighting wave 1 to wave 2.

Mode effects in social surveys – taskforce SASU, Feb Analyses and main results Main result: Size of relative selection effects and pure mode effects of telephone, web and paper for key Safety Monitor, European Social Survey and LFS statistics. Derived results: Relation nature of question and type of population to mode effects Relation between selection and pure mode effect Impact on latent factors and scales deduced from Safety Monitor Evaluation of regular estimation strategies Recommendations for data collection strategy, questionnaire design and estimation strategy

Mode effects in social surveys – taskforce SASU, Feb Planning March to June 2011: data collection July to August 2011: preparation of data analysis files Starting September 2011: Analyses of experimental data Collaboration with Utrecht University until 2013

Mode effects in social surveys – taskforce SASU, Feb Relevance to EU survey on safety Safety Monitor contains questionnaire modules that are very closely related to the new EU survey. Experiment provides useful insight into the expected impact of the survey mode on several safety topics including some topics that will be part of the EU survey. Suggested extensions of study: EU funding to detail analyses to EU relevant topics Conduct similar experiments in a few other EU countries, e.g. through an ESSnet