Mixed-Mode Experiments Rachel Vis, Statistics Netherlands QUEST Ottawa, 24-26 April 2007.

Slides:



Advertisements
Similar presentations
The Response Process Model as a Tool for Evaluating Business Surveys Deirdre Giesen Statistics Netherlands Montreal, June 20th 2007 ICES III.
Advertisements

Market Research Ms. Roberts 10/12. Definition: The process of obtaining the information needed to make sound marketing decisions.
1 Fieldwork logistics and data quality control procedures Kathleen Beegle Workshop 17, Session 2 Designing and Implementing Household Surveys March 31,
1 What the eye doesn’t see: Evaluating a paper based questionnaire using eye-tracking technology Lyn Potaka Statistics NZ.
INTERPRET MARKETING INFORMATION TO TEST HYPOTHESES AND/OR TO RESOLVE ISSUES. INDICATOR 3.05.
Pilot test Group 6. Pilot test We conducted 2 tests with the same person. Test 1: Blindfolded Test 2: User A (one of 4 users in our protocol) Data collecting:
Survey Research Questionnaire construction Types of surveys
Does mode matter? Comparing response burden and data quality of a paper and an electronic business questionnaire. Deirdre Giesen Statistics Netherlands.
Using Market Research. Seeing the Problem Clearly Marketing Research: procedure designed to identify solutions to a specific marketing problem through.
How to Assess Student Learning in Arts Partnerships Part II: Survey Research Revised April 2, 2012 Mary Campbell-Zopf, Ohio Arts Council
8/15/2015Slide 1 The only legitimate mathematical operation that we can use with a variable that we treat as categorical is to count the number of cases.
Literature Direct The New Literature Fulfillment System “In by noon, out by noon.” Visit the web address above and save the.
Hotmail Tutorial This tutorial aims to quickly cover some of the basic elements of web based using msn Hotmail - a free service Use the Index.
Business and Management Research
Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell Chapter 13 Survey Designs.
Regional Gliding School The Flying Scholarship Programmed Instruction Package uses action buttons to make navigation as easy and intuitive as possible.
Interviews By Darelle van Greunen.
Chapter 33 Conducting Marketing Research. The Marketing Research Process 1. Define the Problem 2. Obtaining Data 3. Analyze Data 4. Rec. Solutions 5.
Data and Data Collection Questionnaire
Using the Internet to Conduct Research What Investigators and IRB Members Should Know -- January 29, Lisa Shickle, MS Analyst, VCU Massey Cancer.
FAEIS Project User Opinion Survey 2005 Thursday, June 23, 2005 Washington, D.C. H. Dean SutphinYasamin Miller ProfessorDirector, SRI Agriculture & Extension.
Page 1 LYP Usability Test Results | September 16, 2013 LYP Usability Test Results Conducted by Jayne Schurick Usability Consultant
Mixed mode in LFS: questionnaire design and mode-effects Liisa Larja, Statistics Finland 9th Workshop on Labour Force Survey Methodology Rome, 15 th of.
TRAINING GUIDELINES Do’s and Don’ts for Interview.
Chapter 29 conducting marketing research Section 29.1
Interface Design Natural Design. What is natural design? Intuitive Considers our learned behaviors Naturally designed products are easy to interpret and.
Marketing Research Process Chapter 29. What factors influence restaurants to add low fat menu items? How can they determine success of items? Journal.
Sourcing Event Guide Updated September 12, 2011 Sourcing Event Guide.
Dr D.K. Nyahunzvi Midlands State University 2012.
ACCT How to Use Our Moodle (for students) Click on ‘Onine Learning – Moodle Login’
بِسْمِ اللهِ الرَّحْمنِ الرَّحِيمِِ King Fahd University of Petroleum and Minerals Questionnaire Design presented to: Dr. Abdulaziz A. Bubshait by: Mohammed.
Moodle (Course Management Systems). Surveys and Choices.
Survey Methodology Survey Instruments (1) EPID 626 Lecture 7.
Increasing Efficiency in Data Collection Processes Arie Aharon, Israel Central Bureau of Statistics.
Copyright 2010, The World Bank Group. All Rights Reserved. Testing and Documentation Part II.
Questionnaires Questions can be closed or open Closed questions are easier to analyze, and may be done by computer Can be administered to large populations.
1 Web interviewing at Statistics Netherlands: practices so far Labour Force Survey The Netherlands Björn Janssen Hendrika Lautenbach April 2010.
Welcome! Hello and welcome to English Composition 102, Introduction to Literature. I am looking forward to exploring, writing, and discussing literature.
Consultants attitudes to consumer involvement in clinical research Rachel Thompson, Alan Horwich, Jim Laxton, Joe Flaherty, Andy Norman, Barbara Pearce,
Primary Research Options Interview – One-on-one questions/answers with an expert – Often focuses on open-ended questions – Personal, Phone, Survey.
3.04 Interpret marketing information to test hypotheses and/ or solve issues Marketing Management.
 Reading Quiz  Peer Critiques  Evaluating Peer Critiques.
Navigation Guide for Conveners: Generating Correspondence for the interview Glossary: Requisition - Job Advertisement Hiring Manager – Convener Posting.
1 Cognitive Aspects Associated with Sample Selection Conducted by Respondents in Establishment Surveys La Toya Barnett Thomas Rebecca L. Morrison Grace.
1 Evaluating the User Experience in CAA Environments: What affects User Satisfaction? Gavin Sim Janet C Read Phil Holifield.
Get up to speed Find everyday commands You’ve got Outlook 2007 installed and you’ve taken time to learn about some of the ways it differs from previous.
Copyright 2010, The World Bank Group. All Rights Reserved. DESIGN, PART 2 Questionnaire Design Quality assurance for census 1.
THE MARKETING RESEARCH PROCESS CHAPTER 29 Mrs. Simone Seaton Marketing Management.
Collecting Copyright Transfers and Disclosures via Editorial Manager™ -- Editorial Office Guide 2015.
Section 29.1 Marketing Research Chapter 29 conducting marketing research Section 29.2 The Marketing Survey.
Chapter 29 Conducting Market Research. Objectives  Explain the steps in designing and conducting market research  Compare primary and secondary data.
Component D: Activity D.3: Surveys Department EU Twinning Project.
From Question to Action: Creating In-House Surveys as a part of Data Driven informed Decision Making David Consiglio EDUCAUSE Connect april 22, 2015.
Research Methods for Business Students
Patient Participation meeting Monday 11 February 2013
Microsoft Office Outlook 2003 Lesson 5
Formal Training Survey Tutorial
Evaluation of Research Methods
INFORMAL REPORTS Chapter 9.
Study Island Student Demo:
Study Island Student Demo:
Indiana 4-H Volunteer Orientation
Study Island Student Demo:
Chapter 23 Deciding how to collect data
Help file of PG/VIP Reference
Primary research methods
Study Island Student Demo:
Evaluations and Trials in Alma
Indicator 3.05 Interpret marketing information to test hypotheses and/or to resolve issues.
Presentation transcript:

Mixed-Mode Experiments Rachel Vis, Statistics Netherlands QUEST Ottawa, April 2007

2 Presentation – Introduction mixed-mode research – The test – Results – Future

3 Mixed-mode – primary data collection is expensive  looking for more cost efficient data collection. – drawbacks of MM  mode effects must be minimized. – mode effects can occur during all (?) steps of the statistical process  there should be research on all steps. (prevention and correction)  Q-lab will focus on questionnaire design.

4 First test Aim of the test: – Find the most efficient way to present DK on screen, so that CAWI results are comparable to results in CAPI/CATI – Find the most efficient way to present help- function, so that CAWI results are comparable to results in CAPI/CATI.

5 First test – set up (1) Four versions of ‘don’t know’ 1.DK always on screen as an answer category; 2.DK is never presented; 3.DK is not presented, but if R skips question the option is offered. (with explanation on first screen); 4.DK is presented less visually prominent.

6 First test – set up (2) Two versions of help texts 1.Help text is available by clicking on ?- button. 2.Help text appears when mouse is moved over ?-button.

7

8

9

10

11 First test – set up (3) The questionnaire The test called for a challenging questionnaire. The questionnaire was composed : – out of existing SN survey questions; – of questions needing extra instructions in order to understand them correctly; – of questions to which DK is a possible answer category. There were different versions of the questionnaire.

12 First test – set up (4) Test respondents – The test respondents were recruited from a sample of persons living in towns near the office. – Approximately 120 persons were sent a recruitment letter in which the aim of the test was explained and a participation fee of € 20 was announced. – Respondents could apply by sending back the answer form. – After two weeks all non-respondents were sent a reminder.

13 First test – set up (5) Method – Respondents were invited to come to the office to fill in a questionnaire on the computer. – The respondent was asked to think out loud. – The interviewer observed the respondent while filling in the questionnaire, this was followed by an in-depth interview with follow-up questions. – Everything was recorded on video.

14 Results (1) Recruiting test respondents proved very difficult. From the 120 letters sent, only 12 people responded: 2 did not want to participate. 10 wanted to participate, and 9 actually did.

15 Results (2) – Most respondents indicate that they did not have the wish to answer ‘don’t know’, because they knew the answers to the questions. – One respondent explicitly said that she preferred any answer above ‘don’t know’. – Two persons had wanted to answer ‘don’t know’, but didn’t: In one case the respondent did not think is was an option, since it wasn’t on the screen. She did not try to skip the question and thus did not find out that DK appeared then. In the other case, the respondent did not see the DK- button. She had “forgotten” about it.

16 Results (3) – Most respondents indicate that they did not have the need for extra help. – One person found the ?-button on her own. – Five persons only found the button after a hint. – The respondents indicate that the button is not clear : Two respondents said that they had seen the ?, but had not thought about its purpose. One person was mislead by the place of the button. – There is no difference between the click-button and the mouse-over.

17 Results (4) Though we were not testing navigational issues explicitly, we found the following: – Six people started by asking the interviewer how to move to the next page. – Most respondents used the mouse for selecting an answer and [enter] to move on. – Only a minority used the navigation buttons. – Most respondents indicate that the buttons are too far from the questions.

18 Conclusions Though it was the purpose to test the questionnaire with 30 respondents we concluded that it was best to stop the test. We’re planning a new test for June this year. For this test we will : – develop a new questionnaire; – apply a different layout; – investigate a new way of recruiting test respondents; – include in explanation of the ?-button and DK-button in the introduction.

Questions?