1. An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by NatCen Social Research and TNS BMRB Issues of Coverage, Sampling and Participation in Mixed Mode Surveys Peter Lynn, University of Essex 6 th ESRC Research Methods Festival Oxford, 08-07-2014

2. Mixing Modes “mixing modes gives an opportunity to compensate for the weaknesses of each individual mode at affordable cost” - de Leeuw (2005) Involves an explicit trade-off between costs and (multiple sources of) survey errors Recent interest in mixed modes is particularly stimulated by the (possibly false) notion that the marginal cost of data collection by web is close to zero Lynn | ESRC RMF, 8 July 2014

3. A Distinction Multi-mode (or multiple mode) data collection: Different modes used for different survey items, e.g. - CASI component within a CAPI survey; - CATI follow-up to a mail questionnaire; - etc Mixed mode data collection: The same survey items can be collected by different modes for different sample members: - sequential; - concurrent selective; - concurrent with respondent choice; etc Lynn | ESRC RMF, 8 July 2014

4. Motivation for Mixing Modes Motivation for mixed modes rather than a single mode: Cost reduction; or Coverage/ participation enhancement. (Designs to achieve both simultaneously are proving elusive.) Different motivations tend to suggest rather different designs: Combinations of modes; Sequence of modes (or non-sequential). The coverage, sampling and participation issues may differ between these types of design Lynn | ESRC RMF, 8 July 2014

5. Sequential and Concurrent Mixed Mode Designs Sequential design: Use a number of modes in sequence; Get as much response as possible in one mode, before trying remaining non-respondents in the next mode Cheaper modes first if motivation is to reduce costs Higher response rate modes first if motivation is to maximise participation Concurrent selective design: Offer a different mode to each of 2 or more subsets of sample members Concurrent elective design: Offer each sample member a choice of mode Combination: e.g. Concurrent selective sequential design Lynn | ESRC RMF, 8 July 2014

6. Sequential Design: Example Phase 1: Mail an invite to a web survey to all sample members; Mail a reminder to those who have not responded after a week or two Phase 2: Mail a paper self-completion q’re to those who have still not responded after a further week (or include this with the reminder above) Phase 3: Approach for a face-to-face interview those who have still not responded after a further period Key design choices: Which modes? Which order? Criterion for switching to next phase? Lynn | ESRC RMF, 8 July 2014

7. Coverage Issues with Mixed Mode Surveys I Mixed modes (of approach) can help address frame quality problems. Examples: Inconsistent contact details on sampling frame: only address for some people, only email for others, etc Dual-frame approaches: good coverage is provided only by the union of multiple frames, which have different contact details Note: Mixed modes of approach need not necessarily imply mixed mode data collection Lynn | ESRC RMF, 8 July 2014

8. Inconsistent Contact Details: Example Dutch GPSs (e.g. Labour Force Survey, ESS experiment) Select addresses from Postal Address Register Match phone numbers to addresses (via names) 70% match successfully: Can approach by phone 30% do not match: First approach face-to-face Some of the addresses with phone numbers also require face-to-face follow-up Lynn | ESRC RMF, 8 July 2014

9. Dual-Frame: Example RDD or list-assisted sampling to generate sample of phone numbers: - These numbers are screened to identify households (with phones) Supplementary sample of addresses: - These addresses are screened (face-to-face) to identify households with no (landline) phone The two samples combined give good coverage Lynn | ESRC RMF, 8 July 2014

10. Coverage Issues with Mixed Mode Surveys II Coverage issues can introduce constraints on mixed mode data collection. Example: Desire is a sequential web → face-to-face design Not all sample members are web users Non-web users must skip the web phase, either explicitly or implicitly Lynn | ESRC RMF, 8 July 2014

11. Coverage Issues with Web as a Primary Mode A. Restrict survey to web users: Obvious cost advantages; Non-random under-coverage: requires evaluation and adjustment. Or B. Include non-web users in web mode: Requires provision of hardware, software and training; Various models, e.g. LISS, KnowledgePanel, GIP, ELIPSS. Or C. Include non-web users in a different mode… Lynn | ESRC RMF, 8 July 2014

12. Coverage Issues with Web as a Primary Mode, ctd. Option C, including non-web users in a different mode: May have cost advantages compared to providing equipment – depends on frequency/nature of data collection, etc; May have measurement disadvantages (see next two presentations!) Various designs possible: Web + mail, based on mail-only approach; Interviewer-administered recruitment, followed by web+mail, web+phone, or other mixes; More options are feasible in longitudinal context. Examples: GESIS panel and Gallup panel are both web+mail Lynn | ESRC RMF, 8 July 2014

13. Sampling Issues with Single Mode Web No general population frames with email addresses, therefore first approach must be by a different mode. For frames without names, mail approach requires self- administered respondent selection: This is error-prone if paper-based and may cause dropout if web-based. Interviewer-administration preferred but costly. Alternatively, use non-probability recruitment methods (opt-in panels) and model-based inferential paradigm. Lynn | ESRC RMF, 8 July 2014

14. Sampling Issues with Mixed Mode including Web Frames with partial information can be used (e.g. email addresses for a subset) – but none yet in UK. Instead, single-mode initial approach needed: Mail approach, with web+mail data collection, may offer low- cost solution of reasonable quality in some situations; In UK, this design may work with named-person frames such as admin records, but is hampered by the need for respondent selection for general population surveys; Interviewer approach and respondent selection may be preferable, but only cost-effective for longitudinal surveys. Lynn | ESRC RMF, 8 July 2014

15. Respondent Selection Next / last birthday method: European Social Survey (UK) Experiment ( Villar, 2013 ) Selection compared to birth date info from questionnaire; Approx. 50% correct, 20% incorrect, 30% uncertain (amongst households with 2+ adults) Household roster / grid method: Community Life Survey ( Williams 2013 ) Approx. 25% incorrect selections → Difficult / impossible to control who completes a self- completion questionnaire; → Chance of incorrect respondent may be greater with web Lynn | ESRC RMF, 8 July 2014

16. Participation in Mixed Mode Surveys Response rates in single-mode surveys: Face-to-face > telephone > self-completion – typically Amongst self-completion surveys: Mail > web – often, but not always Composition of response: Broadly similar between modes – typically But some differences Lynn | ESRC RMF, 8 July 2014

17. Participation in Mixed Mode Surveys Population FTF Response Web Response Lynn | ESRC RMF, 8 July 2014

18. Participation in Mixed Mode Surveys Lynn | ESRC RMF, 8 July 2014 Population Mixed Modes Response?

19. Increasing Response Rates Success requires: - All (or most) people who would have responded in mode 1 continue to respond; - Additional people respond too (in mode 2) But it is generally the case that: - (Average) response propensity declines FTF → Phone → Mail → Web - Data collection costs decline in same order - Refusal in one mode reduces propensity to respond in next mode Lynn | ESRC RMF, 8 July 2014

20. Examples of Increasing Response Rates Examples are (all?) from surveys that use a subset of the sequence FTF – Phone – Mail – Web And even then, it is also necessary to “exhaust” each mode E.g. 1 British Household Panel Survey: FTF → phone - Approx. 93% response FTF alone; 96% FTF + phone (amongst previous wave respondents) Lynn | ESRC RMF, 8 July 2014

21. Examples of Increasing Response Rates Examples are (all?) from surveys that use a subset of the sequence FTF – Phone – Mail – Web And even then, it is also necessary to “exhaust” each mode E.g. 1 British Household Panel Survey: FTF → phone - Approx. 93% response FTF alone; 96% FTF + phone (amongst previous wave respondents) E.g. 2 British Crime Survey 2011 Follow-Up: Mail → web - 60% response mail alone; 63% mail + web (amongst BCS respondents) Lynn | ESRC RMF, 8 July 2014

22. Examples of Failing to Increase Response Rates All (?) surveys that use a sequence which is not a subset of FTF – Phone – Mail – Web E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF - 76% response FTF alone; 67% Phone + FTF (w1 resps) Lynn | ESRC RMF, 8 July 2014

23. Examples of Failing to Increase Response Rates All (?) surveys that use a sequence which is not a subset of FTF – Phone – Mail – Web E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF - 76% response FTF alone; 67% Phone + FTF (w1 resps) E.g. 2 UKHLS-IP 2012 wave 5: Web → FTF - 84% response FTF alone; 79% Web + FTF (w4 resps) Lynn | ESRC RMF, 8 July 2014

24. Examples of Failing to Increase Response Rates All (?) surveys that use a sequence which is not a subset of FTF – Phone – Mail – Web E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF - 76% response FTF alone; 67% Phone + FTF (w1 resps) E.g. 2 UKHLS-IP 2012 wave 5: Web → FTF - 84% response FTF alone; 79% Web + FTF (w4 resps) E.g. 3 UKHLS-IP 2012 wave 2: Web → FTF - 85% response FTF alone; 81% Web + FTF (w1 resps) Lynn | ESRC RMF, 8 July 2014

25. Examples of Failing to Increase Response Rates All (?) surveys that use a sequence which is not a subset of FTF – Phone – Mail – Web E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF - 76% response FTF alone; 67% Phone + FTF (w1 resps) E.g. 2 UKHLS-IP 2012 wave 5: Web → FTF - 84% response FTF alone; 79% Web + FTF (w4 resps) E.g. 3 UKHLS-IP 2012 wave 2: Web → FTF - 85% response FTF alone; 81% Web + FTF (w1 resps) E.g. 4 NL-ESS 2009: Web → Phone/FTF (new sample) - 52% response FTF alone; 46% Web + Phone/FTF Lynn | ESRC RMF, 8 July 2014

26. Non-Response Bias Most (of the few) studies to date either found no effect or a modest (assumed) positive effect Hope is that web may disproportionately add young, full-time employed, busy people – who are generally under- represented in surveys But very little evidence either way on this point: - NL-ESS 2009 found almost identical sample composition in the FTF-only and Web + Phone/FTF samples (age, employment, education, etc) - UKHLS 2012 too found no significant differences between FTF-only and Web+FTF (age, gender, household type, etc) Lynn | ESRC RMF, 8 July 2014

27. Mixed mode including Web: UK Examples Two cross-sectional surveys: Community Life Survey ( Williams 2013 ): Random sample of addresses from PAF Advance letter → Mail invitation → Mail reminder → Mail questionnaire 16% responded online + 11% mail = 27% response (no incentive) 19% responded online + 12% mail = 31% response (£5 conditional) 22% responded online + 13% mail = 35% response (£10 conditional) 25% responded online + 14% mail = 39% response (£5 unconditional) European Social Survey (UK) Experiment ( Villar, 2013 ): Random sample of addresses from PAF Advance letter → Mail invitation → Mail reminder → Face-to-face fieldwork 21% responded online + 18% face-to-face = 39% response Lynn | ESRC RMF, 8 July 2014

28. Mixed mode including Web: Longitudinal Examples British Crime Survey re-contact study ( Fong & Williams 2011 ): Issued sample = 30% of BCS respondents who gave an email address; Email invitation → Email reminder → Postal q’re → Postal reminder 35% responded online + 27% by mail = 62% response Lynn | ESRC RMF, 8 July 2014

29. Web+Interviews: Examples 1958 Birth Cohort ( Brown et al 2014 ): 9 th wave (age 55) – 5 years after previous wave Sequential web → telephone 62% responded online + 21% phone = 83% response Understanding Society ( Jäckle et al, 2013 ): Issued sample = wave 5/wave 2 of a household panel Mail(+email) invitation → (email reminders) → Mail reminder → Face-to-face 21% responded online + 55% face-to-face = 76% response (w5 sample) 30% responded online + 50% face-to-face = 80% response (w2 sample; higher incentives) 23% responded online + 55% face-to-face = 78% response (w5 sample, £10) 23% responded online + 51% face-to-face = 74% response (w2 sample, £10) Lynn | ESRC RMF, 8 July 2014

30. Conclusions Potential cost savings from mixed mode may erode if we: -Aim for full population coverage -Aim for response at least as high as could be achieved with single-mode Participation advantages of mixed mode likely to be costly Most promising cost-quality trade-off in the UK currently may be: -Web → Mail → Face-to-face But … measurement concerns (next presentations!) Lynn | ESRC RMF, 8 July 2014

31. An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by NatCen Social Research and TNS BMRB Issues of Coverage, Sampling and Participation in Mixed Mode Surveys Peter Lynn, University of Essex 6 th ESRC Research Methods Festival Oxford, 08-07-2014