1. Panel Response Rates By Ted Vonk - Onderzoekpaleis LONDON 3 October 2006 Advances and opportunities in market research online Dutch Online Panel Study 2006

2. COPYRIGHT This PowerPoint presentation contains the outcomes of the Dutch Online Panel Comparison study 2006 (NOPVO 2006). It is allowed to publish the outcomes of this study if you refer to: The name of the study (NOPVO 2006) The authors: Ted Vonk (Onderzoekpaleis), Robert van Ossenbruggen (ProCression) and Pieter Willems (Millward Brown) Contact: info@onderzoekpaleis.nl

3. Agenda An industry survey: current state of online panel research Response in NOPVO How important is response? Conclusions and recommendations Gastspreker Els Molenaar “Als je wint heb je vrienden”

4. An industry survey: current state of online panel research

5. Some Facts and figures about the Netherlands Demographics Population: 16,3 million 18-65 years:10,5 million Internet penetration: 82% Broadband penetration: 63% (of all housholds, not internet users) Market Research Total market in 2005: € 278 million (UK: € 1.940 million) Spending per capita: € 17 (UK € 32) Online data collection: 25% of revenu quantitative (UK 9%) Over 25 online panels

6. Reasons for strong and fast growth of online research Growing costs of traditional research because of changing laws (social security fees) Decreasing response rates within traditional research Strong growth of internet penetration as a result of an active policy of Dutch government

7. Opinion and market data: over 50% is collected online Today, over 50% of research data is collected by online panels More than 25 online panels are operational in the Netherlands Clients have reacted differently over time 1998 – 2001: Sceptisism Is it respresentative? what are the effects of mode? 2002 – 2004: Fully accepted – hardly any quality questions Speed and costs seem to have convinced all clients 2005 – now: Again increased quality concern large offering, low prices, lack of standards

8. The Dutch Online Panel Study (NOPVO) Why? 1.Provide answer to quality concern 2.Provide guidelines for online panel innovation 3.Unique situation: large number of panels available Research questions NOPVO 1.How do panels work: panel overview 2.Who is the online panel member 3.How do panellists respond? 4.To what extent is opinion and marketing data influenced by panel choice

9. NOPVO method 1.Make inventory of all online panels in the Netherlands 2.Empirical study One survey Every panels draws sample of 1000 respondents Same fieldwork period of 7 day (no reminders) 19 panels participated: Representing in total over 90% of panellists available in NL

10. NOPVO method Extensive data available: Per panel: -Type of incentive -Age of Panel -Invitation policy -etc. Per panel member: -Panel history -Way of recruitment -Number of invitations -Individual response rate -socio demo, etc. Per respondent: -Response (time) -Membership of other panels -data of the questionnaire

11. NOPVO Questionnaire Omnibus study to avoid selective response Labour and transport Political preference Judgement of PM and his image Religion, going to church Moving houses and satisfaction with house and neighbourhood Spontaneous and aided awareness of beer-TV brands Advertising awareness Participation in online research Internet usage Different types of scaling questions, open ended questions etc.

12. Participating panels NOPVO Traditional Full Service part of international chain Internet start upsOnly completed 25 questions of ESOMAR GfK Benelux Intomart GfK Millward Brown Centrum Synovate TNS NIPO Flycatcher Metrixlab Multiscope Netpanel USP-MC CIAO PanelWizard Trendbox Traditional Full ServicePanel Only IBT/Team4 Interview/NSS Motivaction MarketResponse RM Interactive GMI Lightspeed Research SSI (Bloomerce) Panelclix

13. Interesting facts of the panels Majority of panels exists longer than 5 years Different ways of recruiting used Almost all panels give incentives for cooperation Average € 1,20 for 10 minutes Most panels have a restriction policy in inviting panel members for a survey Overview of all details of all panels available at MOAWEB.NL Several panels do not have full details about interview history of panel members

14. Panel response

15. 19.000 panel members invited…. 9.462 completed the questionnaire….. Response of 50%

16. 75% of response is realised within 24 hours TimeThursdayFridaySaturdaySundayMondayTuesdayWednesdayTotal 01.00 - 07.000%4% 2%1%4%5%2% 07.00 - 09.000%9%5%3%5%6%4% 09.00 - 12.000%23%20%21%22%17% 11% 12.00 - 14.001%11%13%17%11%12%13%7% 14.00 - 17.0034%17%20%17% 13%19%26% 17.00 - 20.0028%16% 17%18%16%14%22% 20.00 - 23.0032%13% 18%21% 24%23% 23.00 - 01.005%6%8%5% 12%4%6% Total 50%27%8%6%5%3%1%100%

17. Substantial differences in response per panel Level of response is a result of several factors Panel age Way of recruitment Incentives Panel care Etc. It is hard to determine the effects of a single factor

18. Reasons for high/low response (1) Panels with a high response rate did not only invite their best responding panel members … Panels with a high response rate are mainly new panels Panels with a low response rate are panels that do not clean up their panel database Panels invited panel members that did not complete one questionnaire in over a year despite many invitations Number of panel members is an important communication topic to the market

19. Reasons for high/low response(2) Panel age: Composition of samples to panel age differs a lot per panel New panel members respond better than old panel members Method of recruitment: Recruitment via traditional research (-instruments) gives best response (are willing already) Self registration or via links/banners gives significant lower response Incentives Consistency of incentive is important Positive effect of incentive stops at 1 euro

20. Composition of samples to panel age differs a lot per panel

21. New panel members respond better than old panel members

22. Recruitment methods vary substantially across panels

23. Most panels use a mixture of recruitment methods

24. Self registration or recruitment via links/banners gives significant lower response

25. Certainty about level of incentive is important

26. Positive effect of incentive stops at 1 euro

27. Panel members love to complete questionnaires……

28. Incentives drives young people

29. Some other results Heavy Internet users respond quicker Socio demographics Women respond better than men Older people respond better than younger people Average completion time 12,5 minutes Fastest panel: 11,1 minutes Slowest panel: 15,5 minutes Better respondents respond ……better

30. Better responding panel members respond …… better Average response rate of individual panel members during last 12 months calculated with data about invitations and completes from panel

31. Is a high response rate important?

32. Research outcomes do not depend on response percentage number of words typed in open answer

33. Research outcomes do not depend on response percentage voting in upcoming national elections

34. Research outcomes do not depend on response percentage awareness of beer brands

35. Research outcomes do not depend on response percentage number of words typed in open answer

36. Research outcomes do not depend on response rate Low versus high responding panels No different means on survey questions

37. No response effects explained Non-response effects only occur if there are differences between respondents and non-respondents Non-response effects take place while recruiting panels double opt-in explicit consent to participate in research So, the relevant non-response group has been filtered out before panellists are invited for surveys Potential risk for non-response effects in panels: when response relates to survey topic (selective response)

38. Reponse rate: panel strategy A high response rate does NOT indicate a high sample quality Response rate is a panel strategy component, driven by rules of economics panel setup and maintenance determine response rate the value of response rate is evaluated against recruitment costs

39. Conclusions

40. With online panels, only a specific group of respondents is interviewed. Selection takes places during recruitment, not with inviting panellist for an individual study Therefore, response rate does not reflect sample quality: it reflects a panel business strategy This self selection issue is not new to market research: we’ve seen the same with CATI and P&P Challenges for online panel research are to reach the other group of people

41. Recommendations Deploy other, new recruitment methods Remove badly responding panellists Use online panels for the right kind of research goals use panels for continuous research or trend measurement concept testing but not for incidence rating Do not swap panels in a continuous research project For 0 versus 1 measurement studies, match sample on panel historical data

42. Thank you tedvonk@onderzoekpaleis.nl pieter.willems@nl.millwardbrown.com