© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 1 Quality of Pretesting: Instruments for Evaluation and Standardization Session 23: Survey measurement issues Q2010 in Helsinki May 3-6, 2010

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 2 Contents Pretesting at the FSO Quality standards in qualitative pretesting Future prospects

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 3 Contents Pretesting at the FSO Quality standards in qualitative pretesting Future prospects

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 4 Institutional background Code of Practice (2005), principle 8: “Questionnaires are systematically tested prior to the data collection.” Eurostat QDET (2006): systematic testing in the following cases a new survey new or modified questions additional or modified data collection instrument poor data quality Pretesting:  Increase in data quality  Decrease in respondents’ burden

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 5 Methods of pretesting Quantitative testing methods: Multitude of probands (N > 100) Under field conditions Frequency of problems with the questionnaire Qualitative testing methods: Limited number of probands (N ≤ 20) Under laboratory conditions Reasons for problems with the questionnaire First ideas for improvement  Three step approach

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 6 Step I: Observation Sources of information: Gestures, facial and short verbal expressions (“reality without words”) Remarks in the questionnaire Gain of knowledge: Independent and unaffected behavior without any advance information

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 7 Step II: Cognitive interview Sources of information: Insights in the response process by the use of cognitive methods Narrative description of personal situation Gain of knowledge: Reasons for incorrect or missing answers Individual reality  questionnaire Suggestions for improvement Judgment Comprehension Information Retrieval Response (Tourangeau/Rips/Rasinski 2000)

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 8 Step III: Evaluation of the questionnaire Sources of information: Entries in the questionnaire Remarks, question marks, etc. Gain of knowledge: Actual handling of the questionnaire beyond what respondents thought they had understood

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 9 Contents Pretesting at the FSO Quality standards in qualitative pretesting Future prospects

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 10 Need for quality standards Qualitative methods are often criticized as being unreliable, unrepresentative and insignificant Statistical offices traditionally work quantitatively  new development to elaborate standards for qualitative data and to improve their explanatory power

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 11 Criteria for high quality of qualitative data Checking for generalization without verification Checking for representative probands Checking for researcher effects Triangulation Balancing the evidence (Miles/Huberman 1994)

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 12 Checking: generalization without verification Avoid to regard conclusions for one or two very striking probands as typical (“You see what you want to see.”) Safeguards: Consider positive and negative evidence Quantify qualitative data by the use of QDA software and matrices Double-check codings and conclusions in team

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 13 Checking for representative probands Approximately 20 probands who represent the ordinary respondent in official statistics; group shall be as heterogeneous as possible Safeguards: Select probands adequate for the target population Invite probands with different social background by different ways of recruitment Establish a data base with information on probands

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 14 Checking for effects on probands Intimidated by the test situation Social desirability or acquiescence Concerns about providing information to the “government” Safeguards: Create a comfortable atmosphere Warming-up (course of the test, expectations towards the probands) Underline anonymization and confidentiality

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 15 Checking for effects on interviewer Leading questions Losing distance (”going native“) Safeguards: React in an adequate way, remain neutral Avoid additional remarks on personal opinion or survey question Ask for mutual feedback in team

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 16 Triangulation Confirming results by replicating them Taking different perspectives on the questionnaire Gain an overall picture Safeguards: Data triangulation (probands, places, points in time) Researcher triangulation (teamwork) Methods triangulation (three step approach)

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 17 Methods triangulation Overall picture Observatio n Questionnaire Cognitive interview

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 18 Balancing the evidence “Stronger data can be given more weight in the conclusion.” (Miles/Huberman 1994) Safeguards: Make a note of cases with poor data quality Remember theses cases during data analysis Exclude these cases from the final report, if necessary

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 19 Contents Pretesting in official statistics Selected results Future prospects

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Slide 20 Future prospects Quality standards for qualitative pretesting (e. g. checklists) Online questionnaires Business statistics Elaborated guidelines for cognitive interviewing Exchange of experience between statistical offices

© Federal Statistical Office (FSO), Institute for Research and Development in Federal Statistics Thank you for your attention.