1. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI ESRC Research Methods Festival 8 July 2014, St Catherine’s College, Oxford Understanding the causes of measurement differences by mode Gerry Nicolaas

2. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Contents Background What causes measurement to be different by mode? Question design strategies

3. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Background

4. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Different ways of mixing modes: Mixtures of means of communication Data collection mixtures, e.g. Different data items from the same person Same data items from different people Same data items from the same person at different time points Source: Edith de Leeuw (2005)

5. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Focus of this presentation: Collecting the same data items using different modes Risk of differential measurement error because people may answer questions differently depending on the mode Potential for reduced data comparability

6. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI What causes measurement to be different by mode?

7. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Causes of mode effects (Jäckle et al, 2011) ComprehensionRetrieval JudgmentResponse

8. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Causes of mode effects (Jäckle et al, 2011) Extent of Interviewer involvement Reporting situation Privacy, legitimacy, interaction Willingness to disclose? ► Social desirability bias ComprehensionRetrieval JudgmentResponse

9. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Causes of mode effects (Jäckle et al, 2011) Extent of Interviewer involvement Reporting situation Privacy, legitimacy, interaction Willingness to disclose? ► Social desirability bias ComprehensionRetrieval JudgmentResponse Sufficient effort? ► Satisficing Respondent ability Respondent motivation Task difficulty Extent of Interviewer involvement Visual - Aural stimulus Question format

10. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Causes of mode effects (Jäckle et al, 2011) Extent of Interviewer involvement Reporting situation Privacy, legitimacy, interaction Willingness to disclose? ► Social desirability bias ComprehensionRetrieval JudgmentResponse Sufficient effort? ► Satisficing Respondent ability Respondent motivation Task difficulty Context information Extent of Interviewer involvement Visual - Aural stimulus Question format Respondent control over questionnaire How is question processed? ► Response effects

11. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI To sum up the causes of mode effects: Extent of interviewer involvement Visual versus aural stimulus Question format Respondent control over questionnaire

12. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Mode is more complex than simple distinction between: Face-to-face interview Telephone interview Mail questionnaire Online questionnaire

13. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI For example, face-to-face interview No showcardWith showcard With self- administered module Audio-CASI QuestionAural Visual Visual & Aural recording Response options AuralVisual Visual & Aural recording AnswerOral WrittenTyped

14. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Another example, a telephone interview Traditional Response lists mailed in advance Touchtone Data Entry Interactive Voice Recognition Question Aural (interviewer) Aural (recording) Response options Aural (interviewer) Visual Aural (interviewer) Aural (recording) AnswerOral KeyOral

15. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI A final example, an online questionnaire SimpleWith audio With audio & recording With remote or virtual interviewer QuestionVisualVisual & Aural Response options VisualVisual & Aural AnswerTyped Oral

16. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Mode is a characteristic of a question, not a survey.

17. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Question Design Strategies

18. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Question design strategies: Uni-Mode Construction Writing and presenting questions the same to ensure respondents receive a common mental stimulus Source: Dillman et al (2009); de Leeuw (2005)

19. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Question design strategies: Uni-Mode Construction Writing and presenting questions the same to ensure respondents receive a common mental stimulus Mode-Specific Construction Modifying the question structure, wording or presentation for different modes based on the particular capabilities of each mode Source: Dillman et al (2009); de Leeuw (2005)

20. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Question design strategies: Uni-Mode Construction Writing and presenting questions the same to ensure respondents receive a common mental stimulus Mode-Specific Construction Modifying the question structure, wording or presentation for different modes based on the particular capabilities of each mode Mode-Enhancement Construction Using features not available in all modes to improve quality of responses in one particular mode – minimal evidence Source: Dillman et al (2009); de Leeuw (2005)

21. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Question design strategies: Uni-Mode Construction Writing and presenting questions the same to ensure respondents receive a common mental stimulus Mode-Specific Construction Modifying the question structure, wording or presentation for different modes based on the particular capabilities of each mode Mode-Enhancement Construction Using features not available in all modes to improve quality of responses in one particular mode – minimal evidence Generalised Mode Design Purposively designing questions differently in different modes with the aim of achieving cognitive equivalence – minimal evidence Source: Dillman et al (2009); de Leeuw (2005)

22. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Summing up:

23. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI To sum up: Causes of mode effects: Extent of interviewer involvement Visual versus aural stimulus Question format Respondent control over questionnaire Mode is a characteristic of a question, not a survey

24. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI Thank you Gerry.Nicolaas@ipsos.com | 020 xxxx xxxx 11/11/13 © Ipsos MORIThis work was carried out in accordance with the requirements of the international quality standard for market research, ISO 20252:2006 and with the Ipsos MORI Terms and Conditions which can be found herehere

25. Version 1 | Public (DELETE CLASSIFICATION) Version 1 | Internal Use Only Version 1 | Confidential Version 1 | Strictly Confidential© Ipsos MORI References Dillman, D., Smyth, J. & Christian, L.M. (2009). Internet, Mail and Mixed-Mode Surveys: The Tailored Design Method, 3rd edition. Hoboken, New Jersey: Wiley. Jäckle, A., Lynn, P., Campanelli, P., Nicolaas, G., & Hope, S. (2011). How and When Does the Mode of Data Collection Affect Survey Measurement? ESRA Conference, 21 July 2011. de Leeuw, E. (2005). To Mix or Not to Mix Data Collection Modes in Surveys. Journal of Official Statistics, 21(2), 233-255.