1. Mixed-Mode Experiments Rachel Vis, Statistics Netherlands QUEST Ottawa, 24-26 April 2007

2. 2 Presentation – Introduction mixed-mode research – The test – Results – Future

3. 3 Mixed-mode – primary data collection is expensive  looking for more cost efficient data collection. – drawbacks of MM  mode effects must be minimized. – mode effects can occur during all (?) steps of the statistical process  there should be research on all steps. (prevention and correction)  Q-lab will focus on questionnaire design.

4. 4 First test Aim of the test: – Find the most efficient way to present DK on screen, so that CAWI results are comparable to results in CAPI/CATI – Find the most efficient way to present help- function, so that CAWI results are comparable to results in CAPI/CATI.

5. 5 First test – set up (1) Four versions of ‘don’t know’ 1.DK always on screen as an answer category; 2.DK is never presented; 3.DK is not presented, but if R skips question the option is offered. (with explanation on first screen); 4.DK is presented less visually prominent.

6. 6 First test – set up (2) Two versions of help texts 1.Help text is available by clicking on ?- button. 2.Help text appears when mouse is moved over ?-button.

7. 7

8. 8

9. 9

10. 10

11. 11 First test – set up (3) The questionnaire The test called for a challenging questionnaire. The questionnaire was composed : – out of existing SN survey questions; – of questions needing extra instructions in order to understand them correctly; – of questions to which DK is a possible answer category. There were different versions of the questionnaire.

12. 12 First test – set up (4) Test respondents – The test respondents were recruited from a sample of persons living in towns near the office. – Approximately 120 persons were sent a recruitment letter in which the aim of the test was explained and a participation fee of € 20 was announced. – Respondents could apply by sending back the answer form. – After two weeks all non-respondents were sent a reminder.

13. 13 First test – set up (5) Method – Respondents were invited to come to the office to fill in a questionnaire on the computer. – The respondent was asked to think out loud. – The interviewer observed the respondent while filling in the questionnaire, this was followed by an in-depth interview with follow-up questions. – Everything was recorded on video.

14. 14 Results (1) Recruiting test respondents proved very difficult. From the 120 letters sent, only 12 people responded: 2 did not want to participate. 10 wanted to participate, and 9 actually did.

15. 15 Results (2) – Most respondents indicate that they did not have the wish to answer ‘don’t know’, because they knew the answers to the questions. – One respondent explicitly said that she preferred any answer above ‘don’t know’. – Two persons had wanted to answer ‘don’t know’, but didn’t: In one case the respondent did not think is was an option, since it wasn’t on the screen. She did not try to skip the question and thus did not find out that DK appeared then. In the other case, the respondent did not see the DK- button. She had “forgotten” about it.

16. 16 Results (3) – Most respondents indicate that they did not have the need for extra help. – One person found the ?-button on her own. – Five persons only found the button after a hint. – The respondents indicate that the button is not clear : Two respondents said that they had seen the ?, but had not thought about its purpose. One person was mislead by the place of the button. – There is no difference between the click-button and the mouse-over.

17. 17 Results (4) Though we were not testing navigational issues explicitly, we found the following: – Six people started by asking the interviewer how to move to the next page. – Most respondents used the mouse for selecting an answer and [enter] to move on. – Only a minority used the navigation buttons. – Most respondents indicate that the buttons are too far from the questions.

18. 18 Conclusions Though it was the purpose to test the questionnaire with 30 respondents we concluded that it was best to stop the test. We’re planning a new test for June this year. For this test we will : – develop a new questionnaire; – apply a different layout; – investigate a new way of recruiting test respondents; – include in explanation of the ?-button and DK-button in the introduction.

19. Questions?