1 What happens if you offer a mobile option to your web panel? Results from a pilot study comparing a desktop and mobile device survey Vera Toepoel (Utrecht University, Peter Lugtig (Utrecht University/University of Essex,

2 u According to Statistics Netherlands, 50% of all 12 to 72 year olds with Internet access do this via a mobile device in 2011 u Smartphone penetration rate explosion –In out of 5 people opened their on a smartphone (source: Emerce) Vast increase in mobile phone use

3 u Online probability based panel of Market Response –RDD, Internet population –Panel members ask if they can complete the survey on their mobile phone –Young panel members (age 18-34) have the lowest response probabilities in the panel and are continuously underrepresented in the panel (reasons for non response: don’t feel like it, no time) –Current software not capable of using mobile friendly design u Confirmit software offers ‘responsive design’ (survey automatically optimized for the device chosen) Smartphone as research tool?

4 u June 2012 two surveys with similar sampling frame –(NL, 18+, Internet users, mobile registration): –Standard QSL (current online method) –Confirmit with response design (respondent could chose prefered device) »Soft check for mobile completion Research Design Two parallel surveys Invitation: QSL Confirmit

5 u Amount of people using mobile –Substantial? u Socio-demographic characteristics –Coverage bias: age, edu, gender, urbanization, hh size+composition u Response (break offs and item non-response) u GPS u Evaluation u Response quality (measurement error) –Questionnaire already adapted to mobile (no experiment) –Open-ended questions, check-all-that-apply, primacy Research Questions What happens if you offer a mobile option to your web panel?

6 u 57% used mobile phone throughout the survey (N=443) –252 used mobile (57%) –33 people switched from desktop to mobile during the survey (7%, 12% of mobile starters) –4 people switched from desktop to mobile (1%) –154 people used desktop (35%) –Note: we define mobile as smart phone only. Additional analyses with mobile=all with touchscreen (incl. ipads etc.) produces similar results. Research Questions Amount of people using mobile

7 u Age, hh size and hh composition (kids) significant predictors of mobile use –Older people –Larger households –No kids u No effect of: urbanization, gender, social class, education, household position, family income Research Questions Socio-demographics

8 Hard-to-reach-group Young: mobile increases response probability

9 u Break offs limited –Short questionnaire –4 out of 285 break off on mobile –3 out of 153 on desktop –No evidence that mobile results in higher break offs u Item non-response –7% over 3 questions –No significant difference mobile versus desktop –No significant difference in non-substantive answers (don’t know) Research Questions Response

10 Research Questions time to complete

11 Research Questions Day questionnaire completed

12 Research Questions GPS MobileDesktopTotal GPS ok38%8%25% N %Mobile Home67 Work 14 On the road 11 outside6 abroad2 N=110

13 u n(mobile)=281, n(desktop)=151, N(QSL)=498 u Chi-square test across mobile and desktop not significant u Chi-square test across all three groups not significant Research Questions Evaluation %MobileDesktopTotal confirmit survey qsl Positive (8-10) Neutral (6-7) Negative (1-5) 8987

14 Evaluation Higher evaluation young people

15 Research Questions Response Quality: open-ended questions %MobileDesktop Total (Confirmit) No answer35%33% 0-50 characters52%49%51% charcters 10%16%13% characters 3% Notes: N(mobile)=285, N(desktop)=154 No significant difference

16 Research Questions Response Quality: check-all-that-apply %MobileDesktopTotal (Confirmit) Mean number of shops visited Number of shops visited in past year Notes: N(mobile)=285, N(desktop)=154 No significant differences

17 Research Questions Response Quality: primacy in radio buttons %MobileDesktopTotal Confirmit One a week or more often About once a month About every 3 months About once every 6 month Once a year o less often Don’t know Notes: N(mobile)=285, N(desktop)=154 Frequency of buying clothes in a shop nearby No sigificant difference

18 Feedback mobile option Perfect format, excellent speed! Super! I am glad you figured out that desktop pcs are getting used less often and that questionnaires should be made mobile friendly. I don’t need to get my laptop (evaluation 8: woman, 29) Handy! Better then the desktop since I tend to forget the survey invitations (evaluation 8: woman, 38) Looovvedddd it(evaluation 9: woman, 18) I like this very much. I do not need to get my laptop (evaluation 8: man, 20) Very good!!! I will definetely get higher participation rates if this will be implemented (evaluation 9: man, 32) Don’t like it. I have a pc which I use for all my survey activities. Next time I won’t be doing the survey(evaluation 1: man, 71) Good! I often complete surveys on my mobile phone but this is the first one that reads well, perfect format, excellent speed! Super! (evaluation 9: woman 24) The scrollbar was not working properly. Did not like it. it is easier on desktop(evaluation 3: woman, 58) Good. But it should not take me too long, otherwise I’ll break off(evaluation 8: man, 51)

19 Conclusion u Considerable amount (57%) uses mobile u No effect on non-response u No effect on response quality u Similar evaluation u GPS gives additional insights u No reason to believe that mixed-device is a problem u Able to attract hard-to-reach group such as young people

20 Discussion u What about difficult questionnaires? –Long –grids u What about privacy? u How to offer additional information to researchers? u Coverage bias? Thank you!