1. Thinking bigger and thinking smaller

2. Qualitative Research as a Tool to Improve Interviewer and Questionnaire Quality
Annemiek Luiten: QT, international comparative studies, meta-interviewing instructions, qualiitative data analysis Isje Kuijpers: QT, qualitative field studies, instructing interviewers for QT in the filed Francine Dehue: QT, hypothesis-induced cognitive interviewing, observation of meta-interviews Hans van Kerkoerle: QT, analysis of questionnaire drafts, recruiiment of respondents Math Bosch: Blaise programmimg, voliunteer pool database Ger Snijkers: QT, business survey forms evaluation and tests, Compuer-Assisted Meta-Interviewing, Internet trials Marco Roos: QT, Internet trials

3. Qualitative Research for QuesT: Questionnaire Testing
Improve questionnaire quality collecting meta-data, testing step by step
Examples VDF and POLS
QuesT: “qualitative” <> “quantitative”
QuesT: quality of methods
QuesT: quality of actors
[QuesT for interviewer quality]

4. Qualitative research of questionnaire quality (1) test interviews for meta-data rather than survey data
Meta-information: what the respondent thinks, does and feels while cooperating in a survey
Test interviews: any method to collect respondent related meta-information, e.g.
focus groups & open interviews
‘meta-interviews’ (e.g. ‘cognitive interviews’)
observation (free-format reports/checklists or codes)
evaluation questions (post- or re-interviews, debriefings) 3
1.0 Pretesting programs: definitions

5. Qualitative research of questionnaire quality (2) meta-interviewing techniques
check questions (“Did you count this or that ..?”)
focused probes (“What does ... mean to you in this question/answer?”, “For which months did you ...?”, “What places do you ...?”, Who counts as ..?”, “If yes, which ... do you use?”, “What ...?”, “How many per ...?”, “How did you come to your answer”)
general prompts (Naught-or-nod-prompt, Reassuring prompts, “Can you give an example?”, “Anything more about this?”, “Who, what, where, why, when, how, ....?”)
1.0 Pretesting programs: definitions

6. Qualitative research of questionnaire quality (3) meta-interviewing techniques
vignette tasks/”what if”-classifications (“Would you count an entry on the present form as a vacancy if ...?”)
confidence/sensitivity ratings (“How certain are you about your answer?”, “What effort did it take you to answer this question?”, “How do you feel about this question?”)
paraphrasing tasks (“I just asked you .... Please tell me [in your own words] what the question was [about]?”)
1.0 Pretesting programs: definitions

7. Qualitative research of questionnaire quality (4) meta-interviewing techniques
sorting/ranking tasks
cognitive stimuli: retrospective and concurrent thinking- aloud prompts (general explanation/example followed by prompts: “Please tell me what you think/thought of while ...!” “Just go ahead ...!”, “Go on, go on ...!”, “What do you think right now?”, “Please tell me more ...!”, “Is there anything more that matters?”, ...)
1.0 Pretesting programs: definitions

8. QuestT Model
(1) Definition and feasibility study “What is relevant and measurable? ” => go/no go; prototypes
(2) Qualitative content test
Meta-information on measures
=> less error-prone questionnaire
(3) Qualitative operational test Meta-information on measuring
=> less error-prone data collection
(4) Quantitative pilot: Outcomes => confirmation of design (changes) 4
2.0 What to test when: model

9. Methods in the QuesT Model (1)
Review (Phase 1): (e=..) risk assessment based on (meta-)data from the past accumulated evidence for definition/feasibility
Expert (re-)appraisal (Phases 1-3): (e=..) risk assessment by expert consultation informal testing, expert panels, document design analysis, ...
Focus group (Phases 1-2; respondents): (v=..) n-1 open interview with topic list as agenda, possibly preceded by individual ordinary interviews
Focus group (Phases 1-4; interviewers): (i=..) debriefing sessions, possibly using interviewer observation reports

10. Methods in the QuesT Model (2)
Open interview (Phases 1-2): (v=..) 1-1 open interview with topic list as agenda
Meta-interview (Phase 2): (v=..) 1-1 interview supplemented with meta-interviewing techniques (checks, focused probes, etc.), possibly designed according to risk assessment
Observation (Phases 2-3): (n=..; i=..) meta-data observed during ordinary interviews, possibly using predetermined coding schemes interviewer reports, behavioral or problem coding, informal tests

11. VDF: Vacancies Difficult to Fulfill (1)
Aim evaluate subjective question, develop objective item battery
Blocks 5 government sectors / ‘size’
Goal check feasibility of calibration experiment in future
Factors concepts and their reference sets administrative procedures

12. VDF: Vacancies Difficult to Fulfill (2) QuesT Program
Definition Study e= review and expert appraisal v=15 open interviews
Qualitative Content Studies v=15 open interviews (5) focus groups (1x4, 1x6) n=150 CAPI 73 + CATI 77
test findings prohibiting for calibration study

13. POLS QuesT Program Definition Study e=10 review, expert appraisal
Qualitative Content Studies v=12, i=5 focus groups (2x6, 1x5) v= meta-interviews e= risk assessment, re-formulation
Qualitative Operational Study n= response parameters i= reaction coding, meta-questions, evaluation questions

14. QuesT: to prove or to improve! qualitative <> quantitative
meta-data > substantive data
improve method > produce numbers
catch surprising > describe population test persons
qualitative evidence > statistically as to hypothesized ‘significant’ proof measurement risks
detect unsuspected > theory-driven mismatches search

15. QuesT: trust us if you do not believe us! quality of methods (1)
standard protocols for basic test methods
design next method by preceding one
test the test, prepare the analysis
filter information or go to all depths
introduce the meta-tricks to the layman
check whether the layman gets it
hear what is said afterwards
analyse for progress

16. Meta-data quality (2)
complete, accurate, and relevant answers to meta-questions? well-prepared checks & probes
timing of questions and meta-questions
burden to the respondent
mask the meta- of a question
avoid being ‘cognitive’ more than the population (baseline) likes to be

17. Meta-data quality (3) within-phase multi-method
between-phase multi-method

18. QuesT: do not sample by probability if you probe into mismatches quality of respondents (1)
willingness to take part in a test interview
skills: cognitive, language, social
artificial meta-data
trial-and-error availability, suitability, and ‘risk’

19. QuesT: quality of respondents (2)
Recruitment of test subjects (N=11) Yes No
Probability sample
Subsample from completed survey
Quota sample
Purposive selection from register
Advertisement
Among friends, relations, acquaintances 7 4
(List of a) pool of volunteers
List of former test persons

20. QuesT: professional double-interviewers quality of (meta-)interviewers
motivation and guidance
interaction skills
do the double: meta-data and proper data
do the double: survey and respondent perspective
do the double: procedure and content
subject-matter/questionnaire design/analysis

21. QuesT: quality of double-interviewers (2)
Use of meta-interviewers (N=9) never inc. reg. always
academic
specialized test
specially trained
trained for occasion
less experienced

22. Double conditions of questionnaire testing
Researcher-interviewers (lab and/or field) and field interviewers (special training?!)
Quotas of test subjects (‘sequential’ v) and sampled respondents (n good for ‘p<‘)
Flexible probing and debriefing and systematic response analysis