1. Designing questions for mixed mode data collection: What have we learnt so far? Gerry Nicolaas and Pam Campanelli

2. Research Team National Centre for Social Research (NatCen)  Gerry Nicolaas, Steven Hope, David Hussey, Margaret Blake, Michelle Gray Institute for Soc & Econ Research (ISER)  Peter Lynn, Annette Jäckle, Alita Nandi, Tara Dutt Independent Survey Methods Consultant  Pam Campanelli

3. Background Increasing use of mixed modes  Falling response rates  Rising costs of data collection Risk of reduced data comparability  Coverage error  Non-response error  Measurement error

4. Research Aims & Design Main Objective  Practical advice on how to improve portability of questions across modes Design  A literature review & framework of mixed modes  Mixed Modes experiment (F2F, tel & web)  Cognitive interviews

5. Mode is defined by: Interviewer presence (F2F, tel, none) Oral and/or visual transmission of info  Question delivery  Delivery of response options  Recording of response

6. Face-to-Face Interview A-CASIWritten (CAI)‏Aural & Visual (CAI)‏ SAQ in-int’wWritten (paper)‏Visual (paper)‏ CASI Written (CAI)‏Visual (CAI)‏ FTF (card)‏OralVisualAural FTF (no card)‏OralAural DescriptionResponseResponse Options Question

7. Telephone interview IVROral (CAI)‏Aural (rec)‏ Phone with showcards OralVisualAural TDE Written (CAI)‏ Aural TelephoneOralAural DescriptionResponseResponse Options Question

8. Self-completion A-WebWritten (CAI)‏Aur & Vis (CAI)‏ Web / emailWritten (CAI)‏Visual (CAI)‏ SAQ (e.g. mail)‏Written (paper)‏Visual (paper)‏ DescriptionResponseResponse Options Question

9. Causes of Mode Effects on Measurement (Roberts, Jäckle & Lynn; 2006) ComprehensionRetrievalJudgementResponse Sufficient Effort? Willingness to disclose? Task difficultyR motivationR ability * * Stimulus: Cognitive task Interviewer presence: Pace, non-verbal communication, multitasking Interviewer presence: Anonymity vs. rapport Privacy/legitimacy Shortcutting Social desirability bias

10. Hypotheses (1) Interviewers motivate respondents to make required effort, so less satisficing in interviewer modes compared to self-completion modes  Less acquiescence in F2F & tel than self-completion  Less primacy effects in F2F with showcard than self-compl  Less recency in F2F (no showcard) & tel than primacy in self-completion  Less middle category effect in F2F & tel than self-compl  Less item non-response in F2F & tel than self-completion

11. Hypotheses (2) Interviewers can help respondents with difficult tasks  Ranking questions produce less unusable cases in F2F than in self-completion  End-labelled scales less difficult in F2F & tel than in self- completion

12. Hypotheses (3) Visual stimulus helps cognitive processing of question  Less confusion about direction of end-labelled scales in visual modes than aural mode  Less satisficing in F2F with showcard than F2F without showcard

13. Hypotheses (4) Format effects versus mode effects: 1.Branched questions in telephone versus non- branched questions in F2F interview & web  Differences in responses caused by differences in format rather than mode 2.“Code all that apply” in F2F+showcard & web versus Yes/No questions for each item in tel  Differences in responses caused by differences in format rather than mode

14. Question selection Questions were designed to vary by: Task difficulty Question type  Satisfaction  Other attitudinal  Behavioural  Other factual Sensitivity

15. Mixed Modes experiment Follow-ups to NatCen Omnibus (& BHPS)  Face-to-face, telephone and web comparisons  Experimental design with random allocation Limitation  Restricted to respondents with web access Currently only tel & web data available for BHPS

16. Main findings so far (1) Agree/Disagree questions problematic in any mode but particularly in mixed mode studies Not only does the degree of satisficing vary across modes but also the nature of satisficing Avoid end labelled scales, particularly in tel Ranking questions not equivalent across modes ‘Yes/No’ format not portable across modes Branching questions not equivalent across modes

17. Main findings so far (2) Cognitive interviews can be used to explore mode effects Cognitive interviews raised questions about validity which were not revealed in similar mixed mode studies that only used quant data Cognitive interviews raised questions about methods for detecting satisficing Least preferred mode among cognitive respondents was telephone

18. Selection of initial results Agree/Disagree scales Middle category effect “Code all that apply” versus a series of Yes/No questions for each item Mode preference

19. Agree/Disagree Scales Hypothesis  Agree/Disagree scales are prone to acquiescence bias which is a form of satisficing  So we expect more acquiescence in web than F2F & tel 12 Agree/Disagree questions  3 sets of 4 questions using 5 point scale  Use of opposite statements Initial results from quant analysis of experiment  More acquiescence in F2F & tel compared to web

20. Middle category effects Hypothesis  Choosing middle category can be a form of satisficing  We expect web respondents to be more likely to select middle category than F2F and tel respondents Questions:  2 satisfaction scales, split ballot = 3 vs 7 categories  12 Agree/Disagree questions with 5-point scale Initial results from quant analysis of experiment  Web respondents more likely to choose middle category for 3 & 5 point scales

21. “Code all” vs “Yes/No” questions Hypotheses  Less mode effects between visual and aural modes with Y/N questions than “code all”  No differences when same format is used in all modes Questions  2 sets of 8 items Initial results from quant analysis of experiment  F2F & tel resps more likely than web resps to say “Yes” in series of Y/N questions  Similar effect not found for F2F and web resps in “code all” format

22. What is Cognitive Interviewing? Cognitive Interviewing is an excellent tool for testing survey questions.  It is an in-depth interviewing procedure which pays explicit attention to the mental processes respondents use to answer survey questions.  It uses specialised techniques. “Think-alouds” and “probes” are the most common. For more information, read: Willis, G. (2005), Cognitive Interviewing: A Tool for Improving Questionnaire Design, Thousand Oaks, CA: Sage. Campanelli, P. (2007), Testing Survey Questions, in J. Hox, E. de Leeuw, and D. Dillman (eds), International Handbook of Survey Methodology, Mahwah, NJ: Erlbaum.

23. Cognitive Interviewing Phase Novel because cognitive interviewing... After rather than before quantitative survey (i.e., Omnibus & BHPS survey experiments) Used to explore mode effects (and other Omnibus survey findings) rather than question comprehension, recall, judgment, response All retrospective think alouds and probes after each respondent goes through a short survey questionnaire in CAPI, CATI and CAWI

24. Cognitive Interview Sample (1) Quota sample drawn from Omnibus survey experiment Mode effects are typically at aggregate level But the following individual level behaviours, at the aggregate level, varied by mode:  Agreeing to opposite agree/disagree statements  Misunderstanding a ranking task by giving the same ranking to all or all but one of the items

25. Cognitive Interview Sample (2) Those respondents who agreed to opposite statements and misunderstood the ranking task tended to  have lower levels of education and income  not be working (or if working, in lower level occupations)  be a social renter  be non-white Also included respondents with a contrasting profile The 37 cognitive interviews carried out in respondents’ homes in 8 locations across England and Scotland, lasting approximately 1 hour

26. Sharing 4 of Many Findings Agree/Disgree scales & Acquiescence Other satisficing ‘Yes/no for each’ format Mode preferences

27. Agree/Disagree Scales & Acquiescence Acquiescence response set is tendency to agree to an item regardless of its content Detection of acquiescence: Agreement to opposite statements  individual level see Lenski and Leggett (1960); DeVellis (2003) More agrees in agree/disagree format than in a balanced forced choice format  aggregate level see Schuman and Presser (1981)

28. Cognitive Findings on Agree/Disagree Scales (1) Using the opposite statement method, cognitive interviewing found 23 instances of agreement to opposite statements But more importantly, 21 of these were sensibly justified by respondents

29. Cognitive Findings on Agree/Disagree Scales (2) Example of respondent agreeing to opposite statements for justifiable reasons N36. Compared to other neighbourhoods, this neighbourhood has more properties that are in a poor state of repair. N38. Compared to other neighbourhoods, this neighbourhood has more properties that are well kept. “ In this village,... it’s like half and half. There is a bit [that]... wants doing up and there’s ” the other part which doesn’t (female, no qualifications, very low income, White British)

30. Cognitive Findings on Agree/Disagree Scales (3) Only the remaining two cases problematic Clear acquiescence (possibly due to cultural politeness – see Javeline, 1999 ):  “ I think I don’t understand that, I just say agree ” (female, no qualifications, low income, Pakistani with poor English) Possible acquiescence:  R had ambivalent feelings; found it hard to choose agree or disagree ( female, CSE / O or A Level, low income, White British )  Other respondents with similar views chose the middle category, thus choice of ‘agree’ could be a type of acquiescence

31. Cognitive interview respondents chose middle categories But cognitive interviewing can go beyond the survey findings Middle category satisficing  Cognitive interviewing could distinguish between those who chose the middle category for justifiable reasons and those who appeared to be satisficing Other category satisficing  Cognitive interviewing could distinguish satisficing on other categories Cognitive Findings on Middle Category and Other Satisficing (1)

32. Cognitive Findings on Middle Category and Other Satisficing (2) Examples of possible satisficing: Chose ‘neither nor’ because not that bothered about the state of repair of properties (Female, higher education below degree, medium income, White British) Admitted this is not something she things about (Female, first degree, high income, White British) “ Is slightly satisfied the middle one? I’ll go for the middle one” (Female, first degree, high income, White British)

33. Cognitive Findings on Middle Category and Other Satisficing (3) Examples of clear satisficing: “ I’ll be truthful, I just answered that, with no thought in my head ” (Male, no qualifications, low income, White British) “ To tell you the truth, I just clicked it ” (Female, no qualifications, very low income, White British) “ I’m not too sure, I think you have me on that one.” (Male, CSE / O or A Level, on incapacity benefit, White British)

34. Middle Category and Other Satisficing (4) HypothesisPossible & Clear Middle Category Satisficing CAPICATICAWI 3 category questionsNA02 Agree/Disagree questions203 End-labelled scales26NA Possible & Clear Satisficing on Other Categories CAPICATICAWI 3 category questionsNA10 Rating importance1NA2 End-labelled scales16NA ‘ Yes/No ’ format1NA5

35. ‘Yes/No for each’ versus ‘Code all that apply’ Smyth et al, (2006, 2008) found:  No major issues with ‘yes/no’ format  Lots of advantages (e.g., avoid primacy effects - higher endorsement on all items) Latter study found  No differences between ‘yes/no’ format in phone and web

36. Subtleties going on that could affect aggregate mode comparisons 1.Six instances of clear and possible satisficing 5 of these are in the ‘yes’ category 1 in CAPI and 5 in CAWI 2.Respondents in the middle ground (e.g., qualified their answer or said it depends) 19 of 20 chose ‘yes’ rather than ‘no’ 12 in CAPI and 7 in CAWI 3.Of 6 questions, 2 were more sensitive (increasing income support benefit and redistribution of wealth), slightly more ‘no’ answers in CAWI than in CAPI Cognitive Findings on ‘Yes/No for each’ (1) based on 6 of the questions about reducing poverty in CAPI & CAWI

37. Mode Preferences In mode preference studies, it is typical for the mode the respondent experiences to be the one that the respondent says he or she prefers.  For a household survey example see Groves and Kahn (1979)  For a school based survey example see Brener et al (2006)  For a business survey example see Tarnai and Paxson (2004)

38. Cognitive Findings on Mode Preferences (1) At the end of the cognitive interview, interviewers asked  “Overall what was it like to answer the questionnaire in person versus over the phone versus by yourself on the computer?”  “What did you like or dislike about these 3 different ways you were asked the questions?” To our knowledge no study has used all modes within a respondent.  Should improve the validity of the mode preference data  Possible limitation: Preferences assessed during face-to- face cognitive interviewing

39. Cognitive Findings on Mode Preferences (2) CATI was the least preferred mode. No respondents picked it as their first chose (and only a few picked it as their second choice). Reasons for respondents’ dislike of CATI  Harder to concentrate / can be distracted by other things  Feeling rushed / pressured on the phone  One gives less attention on the phone or has to work harder to think about the question and answer  Phone made the task harder because of the lack of visual stimulus and difficulty in hearing  A general dislike of the phone as a mode of communication

40. Cognitive Findings on Mode Preferences (3) Preferences for CAPI versus CAWI were mixed. This relates to... Respondents level of computer literacy Valuing personal nature of face-to-face or privacy of CAWI Controlling the pace of the CAWI interview, although some respondents felt CAWI made them go too quickly without thinking CAPI leading to better quality data because...  One is more engaged (would “ get more out of her ”)  Answers more considered (more likely to answer and to give a genuine answer)  More committed to the interview and good answers

41. Cognitive Interview Conclusions (1) The pre-planned cognitive interview follow-up was designed to explore mode effects in the Omnibus survey mixed modes experiment Cognitive interviewing can tell us something about mode effects  Through gaining a greater understanding of how mode effects happen

42. Cognitive Interview Conclusions (2) The cognitive findings both do and do not support standard survey research thinking Support: Middle category and other category satisficing:  Although differences are extremely small, there is a consistent pattern: CAWI and CATI respondents appear to be more likely to satisfice than CAPI respondents  Strengthened by respondents comments about different modes

43. Cognitive Interview Conclusions (3) Does not support: Only 2 of 23 respondents who agreed to opposite statements were acquiescing  Casts doubt on practice of using opposite statements to detect acquiescence A lot going on beneath survey findings for ‘yes/no’ format  Casts some doubt that format is problem free

44. Cognitive Interview Conclusions (4) Potential limitations CAPI, CATI and CAWI conditions not fully replicated?  Evidence suggests convincing replicated Respondents still satisficed, interrupted interviewers, and put up with all the distractions in home settings Respondent comments on modes suggests mode differences were felt Retrospective think aloud and probing leading to post hoc rationalisations  Never know for sure, but didn’t appear to be the case

45. Cognitive Interview Conclusions (5) Cognitive interview respondents unusual because  Had been interviewed twice previously  Were quota of respondents with less than optimal behaviour and those with opposite profile  (But quantitative analysis looking at all respondents with those characteristics in the Omnibus survey (n=324) in comparison to the full Omnibus sample show few differences.) Overall, useful research Hope more cognitive interviewing work like this and focusing further on  All of these topics

46. Overall: What have we learnt so far? (1) “Mode” is more complex than simple distinction between face-to-face, telephone, postal, web  Mode relates to a question not a survey  A single question can use more than one channel of communication  Mode interacts with survey question format, e.g., CAWI respondents more likely to choose middle category when scale is short.

47. Overall: What have we learnt so far? (2)  Certain formats less likely to be portable across modes Agree / disagree ‘Yes / No for each’ Branching End-labelled Ranking questions Rating questions

48. Continue with analysis of experimental data Papers  Causes of mode effects on survey measurement  The role of the interviewer in producing mode effects  The role of visual/aural stimuli in producing mode effects  The role of question format in producing ‘mode’ effects  Using cognitive interviews to explore mode effects Training course  Questionnaire design for mixed mode surveys What next?

49. For more details on this project, contact: Gerry.Nicolaas@natcen.ac.uk

51. Response rates Omnibus sample  Face-to-face: 78%  Telephone: 68%  Web: 46% (almost 50% for round 2) BHPS sample  Telephone: 71%  Web: 37%

52. 20 BHPS questions 20 BHPS questions 20 BHPS questions Another 60 questions 20 BHPS questions 20 BHPS questions 20 BHPS questions 20 BHPS questions 20 BHPS questions Another 60 questions Another 60 questions Another 60 questions Another 60 questions Other modules Other BHPS questions NatCen Omnibus BHPS W18 BHPS W19 (12 months after W18) F2F interview after 6 months Tel interview after 6 months Web q’naire after 6 months Tel interview after 6 months Web q’naire after 6 months Cognitive interviews after 6 months 2008 Jan-Jul 2009 Nov-Dec 2009 2010