1. 1 What happens if you offer a mobile option to your web panel? Results from a pilot study comparing a desktop and mobile device survey Vera Toepoel (Utrecht University, V.Toepoel@uu.nl)V.Toepoel@uu.nl Peter Lugtig (Utrecht University/University of Essex, p.lugtig@uu.nl)

2. 2 u According to Statistics Netherlands, 50% of all 12 to 72 year olds with Internet access do this via a mobile device in 2011 u Smartphone penetration rate explosion –In 2011 1 out of 5 people opened their email on a smartphone (source: Emerce) Vast increase in mobile phone use http://www.emerce.nl/research/mobiel-emails-lezen-2011-verdubbeld

3. 3 u Online probability based panel of Market Response –RDD, Internet population –Panel members ask if they can complete the survey on their mobile phone –Young panel members (age 18-34) have the lowest response probabilities in the panel and are continuously underrepresented in the panel (reasons for non response: don’t feel like it, no time) –Current software not capable of using mobile friendly design u Confirmit software offers ‘responsive design’ (survey automatically optimized for the device chosen) Smartphone as research tool?

4. 4 u June 2012 two surveys with similar sampling frame –(NL, 18+, Internet users, mobile registration): –Standard QSL (current online method) –Confirmit with response design (respondent could chose prefered device) »Soft check for mobile completion Research Design Two parallel surveys Invitation: QSL Confirmit

5. 5 u Amount of people using mobile –Substantial? u Socio-demographic characteristics –Coverage bias: age, edu, gender, urbanization, hh size+composition u Response (break offs and item non-response) u GPS u Evaluation u Response quality (measurement error) –Questionnaire already adapted to mobile (no experiment) –Open-ended questions, check-all-that-apply, primacy Research Questions What happens if you offer a mobile option to your web panel?

6. 6 u 57% used mobile phone throughout the survey (N=443) –252 used mobile (57%) –33 people switched from desktop to mobile during the survey (7%, 12% of mobile starters) –4 people switched from desktop to mobile (1%) –154 people used desktop (35%) –Note: we define mobile as smart phone only. Additional analyses with mobile=all with touchscreen (incl. ipads etc.) produces similar results. Research Questions Amount of people using mobile

7. 7 u Age, hh size and hh composition (kids) significant predictors of mobile use –Older people –Larger households –No kids u No effect of: urbanization, gender, social class, education, household position, family income Research Questions Socio-demographics

8. 8 Hard-to-reach-group Young: mobile increases response probability

9. 9 u Break offs limited –Short questionnaire –4 out of 285 break off on mobile –3 out of 153 on desktop –No evidence that mobile results in higher break offs u Item non-response –7% over 3 questions –No significant difference mobile versus desktop –No significant difference in non-substantive answers (don’t know) Research Questions Response

10. 10 Research Questions time to complete

11. 11 Research Questions Day questionnaire completed

12. 12 Research Questions GPS MobileDesktopTotal GPS ok38%8%25% N253190443 %Mobile Home67 Work 14 On the road 11 outside6 abroad2 N=110

13. 13 u n(mobile)=281, n(desktop)=151, N(QSL)=498 u Chi-square test across mobile and desktop not significant u Chi-square test across all three groups not significant Research Questions Evaluation %MobileDesktopTotal confirmit survey qsl Positive (8-10) 42444338 Neutral (6-7) 50474955 Negative (1-5) 8987

14. 14 Evaluation Higher evaluation young people 7.2 18-24 25-34 35-44 45-54 55-64 65+ 7.7 7.1 7.5 7.1 6.9 6.4

15. 15 Research Questions Response Quality: open-ended questions %MobileDesktop Total (Confirmit) No answer35%33% 0-50 characters52%49%51% 51-100 charcters 10%16%13% 100 + characters 3% Notes: N(mobile)=285, N(desktop)=154 No significant difference

16. 16 Research Questions Response Quality: check-all-that-apply %MobileDesktopTotal (Confirmit) 024.221.423.2 19.19.79.3 215.418.216.4 314.712.313.9 414.415.614.8 58.88.48.7 67.06.56.8 74.64.54.6 81.83.22.3 Mean number of shops visited 2.72.8 Number of shops visited in past year Notes: N(mobile)=285, N(desktop)=154 No significant differences

17. 17 Research Questions Response Quality: primacy in radio buttons %MobileDesktopTotal Confirmit One a week or more often 3.90.62.7 About once a month 23.927.725.2 About every 3 months 37.232.135.4 About once every 6 month 21.426.423.2 Once a year o less often 10.29.49.9 Don’t know3.53.83.6 Notes: N(mobile)=285, N(desktop)=154 Frequency of buying clothes in a shop nearby No sigificant difference

18. 18 Feedback mobile option Perfect format, excellent speed! Super! I am glad you figured out that desktop pcs are getting used less often and that questionnaires should be made mobile friendly. I don’t need to get my laptop (evaluation 8: woman, 29) Handy! Better then the desktop since I tend to forget the survey invitations (evaluation 8: woman, 38) Looovvedddd it(evaluation 9: woman, 18) I like this very much. I do not need to get my laptop (evaluation 8: man, 20) Very good!!! I will definetely get higher participation rates if this will be implemented (evaluation 9: man, 32) Don’t like it. I have a pc which I use for all my survey activities. Next time I won’t be doing the survey(evaluation 1: man, 71) Good! I often complete surveys on my mobile phone but this is the first one that reads well, perfect format, excellent speed! Super! (evaluation 9: woman 24) The scrollbar was not working properly. Did not like it. it is easier on desktop(evaluation 3: woman, 58) Good. But it should not take me too long, otherwise I’ll break off(evaluation 8: man, 51)

19. 19 Conclusion u Considerable amount (57%) uses mobile u No effect on non-response u No effect on response quality u Similar evaluation u GPS gives additional insights u No reason to believe that mixed-device is a problem u Able to attract hard-to-reach group such as young people

20. 20 Discussion u What about difficult questionnaires? –Long –grids u What about privacy? u How to offer additional information to researchers? u Coverage bias? Thank you!