The measurement effect in PC smartphone and tablet surveys Valerija Kolbas University of Essex ISER Ipsos-MORI The European Survey Research Association Conference July, 2015 Reykjavik

Background and motivation Limiting surveys to the PC mode affects the size and representativeness of the sample PC, smartphones, tablets differ in:  size of the screen  input method  speed of connection  processing power Different effect on the measurement error

Background and motivation Smartphone  more break-offs  longer completion time  More straightlining Mixed results Other indicators  question order effect  primacy effects  open-ended answers  response distribution … but non-optimized mobile design affects completion rates, satisfaction with the survey  Tablet  Less primacy effect  Less straightlining  Less or comparable completion tim e

Background and motivation PC and mobile response distributions are equally affected by response formats Drop-boxes –preference for first options Grids – preference for visible options, straightlining - no conclusive evidence which format is better for mobiles

Research Question How mode of administration and response format affect survey responses Indicators of measurement error:  overall satisfaction rates  straightlining  response distribution  length of open-end questions

Survey and Questionnaire design National Satisfaction Survey 2014 Administered to final year higher education students in UK 22 core questions using a 5-point Likert scale 2 open-end questions Mixed-mode: self-selected mail, phone, web 5 response formats – randomly allocated

Sample composition PC N=5529 Smartphone N=3196 Tablet N=551 Radio-button Drop-box () Drop-box (0) Drop-box (-) Drop-box (+) Radio-button Web Survey Sample N=9276

Screenshots: response design Once clicked a list with options appears on a separate screen. Responses always in the same order

Screenshots: response design All questions are visible on the screen. Requires horizontal scrolling. Portrait or landscape viewin g. Requires vertical scrolling.

10 Measurement Effect between PC smartphone tablet responses Comparisons made across all three modes, but within one radio-button response format

Measurement Effect between PC smartphone tablet FpPCsmartphonetablet Straightlining3.9<.056.9%10.3%6.4% Modal responses <1>.05 ‘Mostly Agree’ ‘Definitely Agree’ & ‘Mostly Agree’ Mean values Positive feedback 1.3> Negative feedback 1> MANOVA to test for differences F=2.3, p<.05

ME between PC, smartphone, tablet summary Smartphone straightlining significantly higher Tablet straightlining rate the lowest Signs of the visibility effect for smartphone Other quality indicators comparable between all three modes

13 Measurement Error between different response designs presented on a smartphone and a tablet Comparisons made across five survey response formats within smartphone and tablet modes

MANOVA test of differences F=2.7, p<.05 Fp device2.7<.05 format1.7<.05 interaction  Both device and format affected data quality

Responses in Drop-box with a positive initial option Smartphones  higher selection of initially suggested response.  less moderately positive responses  11.1% straightliners  Mean 4.3 Tablets  Similar selection of extreme and moderately positive responses  More negative responses  5.6% straightliners  Mean 4.1

Responses in Drop-box with a negative initial option Smartphones  More extreme negative responses  Less positive responses  8.4% straightliners  Mean 4 Tablets  More positive responses  Extremely low ‘Definitely Disagree’ frequency  2.5% straightliners  Mean 4.1

Responses in Drop-box with a middle initial option Smartphone and Tablet Comparable results  Weak evidence of selecting the middle option  Most frequent ‘Mostly Agree/Disagree’ selection across formats  6.5% and 6.1% of straightliners  Mean 4 an 3.9

ME indicators summary Smartphone – answers affected by response formats.  Initially suggested response is selected more often Tablet – no strong effect of response formats. No significant differences in the length of open answers between formats.

Potential Limitations No reverse-coding Similar question wording Not counterbalanced question or response order Instructions universal for each response format Self selected device condition Survey sample: highly educated, IT literate, similar age, highly motivated

Thank You