Effects of Design in Web Surveys Vera Toepoel Tilburg University The Netherlands 1/17/2019

CentERdata: Two Online Panels 1. CentERpanel:l Exists for 17 years 2000 households Respondents fill out questionnaires every week Online interviews as method, but: Probability sample drawn from address sampling frame of Statistics Netherlands Recruitment of new panel members address-based Includes households without internet access (less than 20%): Equipment 1/17/2019

CentERdata: Two Online Panels 2. LISS Panel Grant from The Netherlands Organisation for Scientific Research 5000 households Established in 2007 (we fielded 1st questionnaire!) Respondents fill out questionnaires every month Online interviews as method, but: Probability sample drawn from address sampling frame of Statistics Netherlands Contacted by letter, telephone or visit Includes households without internet access (less than 20%): Equipment 1/17/2019

1 item per screen 1/17/2019

4 items per screen 1/17/2019

10 items per screen 1/17/2019

1/17/2019

Answer categories 1/17/2019

Open-ended 1/17/2019

Vertical: positive to negative 1/17/2019

Horizontal 1/17/2019

Numbers 1 to 5 1/17/2019

Numbers 5 to 1 1/17/2019

Numbers 2 to -2 1/17/2019

Trained Respondents: Panel conditioning Content (knowledge on topics) Prepare for future surveys Develop attitudes Procedure (question-answering process) Learn how to interpret questions Answer strategically Speed through the survey 1/17/2019

Procedure (answer process) Differences between trained and fresh respondents with regard to web survey design choices Items per screen Response category effects Question layout 1/17/2019

Overall: Difference in mean duration of the entire survey between panels: 436 seconds for the trained panel and 576 seconds for the fresh panel. 1/17/2019

Experiment 1: Items per screen Social Desirability Scale 10 items 3 different formats: 1 item per screen 5 items per screen 10 items per screen 1/17/2019

Experiment 1: Items per screen Trained respondents had higher inter-item correlations for multiple-item-per-screen formats. No significant difference in item non-response. Mean score of the Social Desirability Scale showed no evidence for social desirability bias. The mean duration to complete the ten social desirability items did not differ significantly between panels. 1/17/2019

Experiment 2: Answer Categories 1/17/2019

Experiment 2: Answer Categories 1/17/2019

Experiment 2: Answer Categories Category effect found No difference in category effect between trained and fresh respondents 1/17/2019

Experiment 3: Question Layout Question: Overall, how would you rate the quality of education in the Netherlands? Answer: 5-point scale Six formats: Reference format (decremental) Reverse scale: incremental Horizontal layout Add numbers 1 to 5 to verbal labels Add numbers 5 to 1 to verbal labels Add numbers 2 to -2 to verbal labels 1/17/2019

Experiment 3: Question Layout Decremental vs. incremental: T+ F Vertical vs. horizontal layout: - No numbers vs. numbers 1 to 5:- Numbers 1 to 5 vs. numbers 5 to 1: T+F Numbers 5 to 1 vs. Numbers 2 to -2: T+F Trained respondents more easily selected one of the first options. T=significant differences in Trained panel F=significant differences in Fresh panel 1/17/2019

Design Effects in Web Surveys: Comparing Trained and Fresh Respondents Overall little differences between trained and fresh respondents Trained respondents are somewhat more sensitive to satisficing: Shorter completion times Higher inter-item correlations for multiple-items-per-screen formats Select first response options more often 1/17/2019

Current and Future Research It has been little more than a decade since systematic research was begun on visual design effects in web surveys. In the last decade dozens of studies have been conducted It is now important that we begin to understand the importance of each of the visual effects Can we reduce visual effects by effective question writing?! 1/17/2019

Effective Question Writing Tourangeau, Couper, and Conrad (POQ 2007) suggest there may be a hierarchy of features that respondents attend to: Verbal language>numbers> visual cues Question: Can the effects of visual layout be diminished through greater use of verbal language and numbers? 1/17/2019

Experiment 1: Visual Heuristics (joint with Don Dillman) Tourangeau, Couper, and Conrad (POQ 2004; 2007): Middle means typical: respondents will see the middle option as the most typical Left and top means first: the leftmost or top option will be seen as the ‘first’ in conceptual sense Near means related: options that are physically near each other are expected to be related conceptually Up means good: the top option will be seen as the most desirable Like means close: visually similar options will be seen as closer conceptually Experimental conditions: Polar point or fully labeled scale With or without numbers (1 to 5) 1/17/2019

Middle Means Typical Fully labeled: even spacing Fully labeled: uneven spacing 1/17/2019

Left and Top Means First Fully labeled with color: consistent ordering Fully labeled with color: inconsistent ordering 1/17/2019

Near Means Related Polar point with numbers: separate screens Polar point with numbers: single screen 1/17/2019

Up Means Good Polar point with numbers: incremental Polar point with numbers: decremental 1/17/2019

Like Means Close Polar point Polar point with color 1/17/2019

Like Means Close Polar point with numbers (1 to 5) Polar point with different numbers (-2 to 2) 1/17/2019

Labels, numbers and visual heuristics: is there a hierarchy? 1. Middle Means Typical 2.Left and Top Means First 3. Near Means Related 4. Up Means Good 5. Like Means Close Effect heuristic? no yes Numbers reduced effect? Color: yes Dif. #: no Effect heuristic in fully labeled scales? 1/17/2019

Experiment 2: Pictures in web surveys (joint with Mick Couper) Replicate study Couper, Tourangeau, and Kenyon (POQ 2004) 1. No Picture 2. Low frequency picture 3. High frequency picture Add verbal instructions A. No verbal instruction B. Instruction to include both high and low frequency instances C. Instruction to include only low frequency instances 1/17/2019

Low and High frequency picture 1/17/2019

Can verbal instructions reduce the effects of pictures? MANOVA main effect instructions lambda=.597, p<.0001  main effect pictures lambda=.964, p<.0001 interaction instructions*pictures lambda=.9691,p<.0001 This suggests that while both the main effect and interaction are significant, instructions explain more of the variation in the answers than pictures! 1/17/2019

Future Research How to reduce visual design effects in web surveys 1/17/2019

17-1-2019 LISS data Every researcher (irrespective of nationality) who wants to collect data for scientific, policy or societal relevant research can collect data via the LISS panel at no cost Proposals can be submitted through www.lissdata.nl Existing data free available for academic use longitudinal core studies proposed studies disseminated through www.lissdata.nl 1/17/2019