1. Online Opinions – A Pilot to Extend ONS Social Data Collection Capabilities Ed Dunn ONS Social Survey Division

2. Overview Background and context setting The pilot study; our key objectives and design Findings Next steps

3. Pilot Key Objectives – ‘Dipping our Toes’ Need to examine web collection Investigate unit and item response rates Investigate respondent characteristics Evaluate the hardware/software capabilities Examine the effect of offering an incentive Design and administration of web surveys Recommendations for further work Deliver to a short timescale (by December 2008)

4. Sample Design and Respondent Selection – Web Pilot Simple random sample of 2,000 households from English Postal Address File Advance letter Half offered a £10 Amazon gift certificate to take part online Letter asked adult with most recent birthday to complete the survey Unique username and password provided to access survey from ONS website Follow-up letter

5. Sample Design and Respondent Selection- Opinions Survey (OPN) OPN is multi-stage cluster design Respondent selection via KISH grid November OPN sampled 2,010 households Book of postal stamps sent with advance letter

6. Questionnaire Design (1) Based upon ONS Opinions Survey (OPN) Used question blocks equivalent to a ‘normal’ month of OPN Questions reviewed for suitability for web Some modifications necessary Questionnaire tested internally

7. Questionnaire Design (2) Design and layout reviewed for web survey* Page Design Next/Back Button Progress Bar * See Dillman, D.A (2007) Mail and internet surveys - the tailored design method (2nd edition) J Wiley & Sons, New Jersey

8. Questionnaire Design (2)

9. Unit Response Internet pilot 364 fully complete responses received (18%) 32 partial responses (2%) Little difference between incentive (18%) and non- incentive (19%) group But, Full sample = 2000 10% unknown eligibility and only 61% have internet access at home (LCF) Available sample = 1098 New Response = 33% complete responses November OPN 59% overall response (with known eligibility)

10. Metrics 80% of respondents completed web survey in under 25 minutes 75% completed 8am-6pm Most popular hours to begin survey were 4pm-5pm (10%) and 7pm-8pm (11%) 25% completed on weekend Wednesdays and Fridays most popular OPN face to face takes c.45mins (excl. admin time)

11. Results (1) Weighted results; OPN routinely weighted but pilot required some methodological work. OPN was then re-weighted to same population totals. 2-tailed t test applied to differences Some clear differences in profile of internet pilot respondents compared to OPN and key question responses

12. Results (2) The pilot respondents were more likely to be: Aged 25-44 and 55-64 Married and living with partner White Better educated (with degree level qualification) Managers or supervisors In good health

13. Results (3) In terms of key question responses pilot respondents were less likely to: Smoke Have a disability Think charities played an important role in society Think HM Revenue & Customs treated them fairly Pilot respondents were more likely to: Eat healthily (5 portions of fruit or vegetables a day)

14. Next Steps Limitations of this pilot widely accepted Pilot has demonstrated viability of conducting a major ONS social survey online and receiving a significant, if minority, response Pilot has highlighted logistical and technological issues Pilot has highlighted risk of significant bias Identified areas for further work via more sophisticated pilot designs Plan to run a further pilot on OPN in 2010 and pilot on LFS early in 2011

15. References Dillman, D.A. 2007. Mail and Internet Surveys – The Tailored Design Method (2 nd ed). J Wiley and Sons, New Jersey Flatley, J. 2001. The Internet as a Mode of Data Collection in Government Social Surveys: Issues and Investigation Social Survey Methodology Bulletin, ONS 49(7) Fricker, R and Schonlau, M. 2002. Advantages and Disadvantages of Internet Research Surveys: Evidence from the Literature. Field Methods 14(4) Solomon, D. 2001. Conducting Web-Based Surveys. Practical Assessment, Research and Evaluation 7(19)

16. Internet and the UK LFS Dean Fletcher UK ONS

17. Internet and the UK LFS Current Modes: Wave 1 Face to Face Wave 2-5 TU or ftf Response on UK LFS all time low 65% wave 1, 57% overall. Achieved sample size decreasing or much more costly to obtain Efficiency savings face to face costs 3 times TU Pilot shows some groups with current high non- response will take part in the internet. TU costs 3 times Internet.

18. Internet and the UK LFS Pilot showed technology exists. 4 main Problems: Collecting close to the Reference week Problem of integrating the internet with the other modes to prevent bias Mode effects Collecting from Households not persons

19. Face to Face W1 TU W2-5 Internet and the UK LFS Analysed other approaches but derived duel sample approach after considering results of the pilot Initial Sample Registered for internet Respond to internet FtF W1 TU W2-5

20. Internet and the UK LFS Currently reviewing questionnaire for internet suitability based on pilot results and others research to minimise mode effects. Household collection seems feasible Use Confirmit to test ideas Use BLAISE IS? To run pilot in 2011 in a few NUTS2 areas If successful run full design by 2012 Modify collection by 2013

21. Internet and the UK LFS Issues Internet pushes up refusals Unequal response Changes to internet over time

22. Internet and the UK LFS Also monitoring wave 2-5 work by other NSi’s to implement if shown to be successful But current systems do not allow internet, TU and ftf integration. Cost savings and non-response mainly achieved for UK at Wave 1.

23. Internet and the UK LFS So far we have had: OPN Pilot Input at International Workshop Prof. Mick Couper’s work and training Literature review But interested to have design scrutinised for issues.

24. Internet and the UK LFS Concluding remark UK Can’t afford not to implement an internet collection mechanism if possible in the next 2 years.