1. Margaret Blake RESEARCH METHODS FESTIVAL: Evaluating questionnaires with cognitive testing

2. How do we design and administer questions in surveys in a way that will deliver valid and reliable data?

3. The design of survey questions is crucial You need good quality questions to produce good quality data But what makes a good survey question and how will we know if it will produce valid and reliable data?

4. What makes a good question? Research question and objective of survey question clearly defined Concept(s) to be included in the question are clear Research questions and concepts converted into survey questions which follow principles of good questionnaire design Question has been evaluated or tested

5. Evaluating questions Desk based evaluations Expert panels Standard field pilots and dress rehearsals Respondent debriefing Analysis of existing data Split ballot experiments Behaviour coding Interviewer rating Cognitive testing

6. The question and answer model

7. Cognitive testing in practice Small sample to reflect range and diversity of population Administer survey question/ instrument being tested Collect verbal reports from respondent Think aloud Probing (scripted and spontaneous) Analysis of verbal reports as qualitative data Reporting of problems with questions Recommendations of changes to questions

8. Youth in Europe Cross national project with testing in four countries Testing young people and parents Sample to include gender and ethnic mix Self-completion questionnaires Use of concurrent probing and think aloud Focus on problems with questions Constraints of comparability across countries Questions will be translated within each country

9. Variations Testing in multiple languages Test different types of instruments Diaries Card sorts Consent to data linkage Letters or other survey materials Use vignettes Test instruments in different modes Look at the question and answer process rather than testing questions

10. Mixed modes project Not testing questions Looking for evidence of mode effects and exploring how and why they occur Sample based on response characteristics in survey experiment, not on demographics Testing three modes in one interview Simulation of modes in face to face interview Retrospective think aloud Absence of spontaneous probing

11. To finish Cognitive testing offers something different There are limitations and constraints Cognitive testing can be used for more than question testing