Margaret Blake RESEARCH METHODS FESTIVAL: Evaluating questionnaires with cognitive testing.

Slides:

Advertisements

Similar presentations

Survey design. What is a survey?? Asking questions – questionnaires Finding out things about people Simple things – lots of people What things? What people?

Advertisements

1 COMM 301: Empirical Research in Communication Kwan M Lee Lect4_1.

Children’s subjective well-being Findings from national surveys in England International Society for Child Indicators Conference, 27 th July 2011.

Collecting data Chapter 5

IBD PATIENT PANELS IBD Patient Panel Surveys Why Carry Out a Survey?  Bring about improvements in part of a service that you think may not up to standard.

Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.

RESEARCH METHODS Lecture 24

Research Methods Overview of quantitative and qualitative methods.

Survey Design Steps in Conducting a survey.  There are two basic steps for conducting a survey  Design and Planning  Data Collection.

Analysing and interpreting cognitive interview data: a qualitative approach.

Chapter 13 Survey Designs

E 43 Johan Brink, IIE 22 November Quantitative research Lecture

Objectives of Session Seven Complete in-class survey Discuss question formats and ordering Case study: face-to-face interviews v. self-administered questionnaires.

Developing a Questionnaire Chapter 4. Using Questionnaires in Survey Research Construction is key to valid and reliable research Well written and manageable.

Chapter 13 Survey Designs

Designing Questionnaires for Research By Janna McColgan EDUC 491/EDUC 403 Research Spring 2012.

Survey Designs EDUC 640- Dr. William M. Bauer

Copyright c 2001 The McGraw-Hill Companies, Inc.1 Chapter 9 Surveys, Questionnaires, and Polls Most commonly used quantitative method Used for obtaining.

Sara Xiong & Rebecca Radle, Advisor: Dr. Susan Wolfgram, University of Wisconsin-Stout Research Question & Hypothesis What resources do young single parents.

CAHPS Overview Clinician & Group Surveys: Practical Options for Implementation and Use AHRQ ANNUAL MEETING SEPTEMBER 18, 2011 Christine Crofton, PhD CAHPS.

Conducting a Job Analysis to Establish the Examination Content Domain Patricia M. Muenzen Associate Director of Research Programs Professional Examination.

How a media industry identifies audiences for its products.

Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell Chapter 13 Survey Designs.

Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell Chapter 13 Survey Designs.

Surveys and Questionnaires Unit 4 Online Tutorial 2.

MEADOW: Guidelines for a European survey of organisations Nathalie Greenan CEE and TEPP-CNRS Exploring possibilities for the development of European data.

Ethno-cultural groups in Population Censuses An evaluation of the UNECE/EUROSTAT Recommendations for Population Censuses and proposals for the 2010 round.

Orientations of Young Men and Women to Citizenship and European Identity

Understanding Society: the UK Household Longitudinal Study Professor Vernon Gayle ISER, Essex University

Surveys and Questionnaires Online Tutorial. What is a survey? A survey is a method of gathering information from a number of individuals, known as a sample,

Action Research March 12, 2012 Data Collection. Qualities of Data Collection  Generalizability – not necessary; goal is to improve school or classroom.

SPECA Regional Wrokshop on Disability Statistics, Dec 13-15, 2006 Issues Related to Disability Measurement: Cognitive testing and mode Jennifer Madans.

Key steps in conducting survey research Decide if a survey is the best design to use Short time, economical, dispersed population, anonymity Report what.

Data gathering. Overview Four key issues of data gathering Data recording Interviews Questionnaires Observation Choosing and combining techniques.

Evaluating a Research Report

Chapter 29 conducting marketing research Section 29.1

Quality Assessment July 31, 2006 Informing Practice.

NAME Evaluation Report Name of author(s) Name of institution Year.

Research Seminars in IT in Education (MIT6003) The use of computers in educational research Dr Jacky Pow.

1 Learning Objectives: 1.Understand data collection principles and practices. 2.Describe the differences between collecting qualitative and quantitative.

How do we assess whether we are improving instrument design? Alice McGee.

Designing Survey Instruments. Creating a Survey Instrument  Survey instruments should help researchers collect the most accurate data and reach the most.

VALIDITY AND VALIDATION: AN INTRODUCTION Note: I have included explanatory notes for each slide. To access these, you will probably have to save the file.

CSCI 4163 / CSCI 6904 – Winter Housekeeping  Clarification about due date for reading comments/questions  Skills sheet  Active listening handout.

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.

Evaluating Survey Items and Scales Bonnie L. Halpern-Felsher, Ph.D. Professor University of California, San Francisco.

Survey Methodology Survey Instruments (1) EPID 626 Lecture 7.

Self-completion Questionnaires

© 2006 by The McGraw-Hill Companies, Inc. All rights reserved. 1 Chapter 9 Surveys, Questionnaires, and Polls Most commonly used quantitative method –Used.

Conducted on behalf of: Conducted by: In Association With: PTV DataSource GeoStats PB Chandra Bhat Mark Bradley Mary Kay Christopher Keith Lawton Survey.

AS Sociology.  Structured  Unstructured  Semi structured  Focus groups WE.

Asking users & experts. The aims Discuss the role of interviews & questionnaires in evaluation. Teach basic questionnaire design. Describe how do interviews,

Stretching Your Data Management Skills Chuck Humphrey University of Alberta Atlantic DLI Workshop 2003.

Development Team Day 5a October Aim To explore approaches to evaluating the impact of the curriculum on pupil learning.

Data gathering (Chapter 7 Interaction Design Text)

Measurement and Health Information Systems Working Paper No.8 11 November 2005 STATISTICAL COMMISSION andSTATISTICAL OFFICE OF THE UN ECONOMIC COMMISSION.

Copyright © 2008 by Pearson Education, Inc. Upper Saddle River, New Jersey All rights reserved. John W. Creswell Educational Research: Planning,

Lesson 11: Designing Research. Naturalistic Observation When designing a naturalistic observation researchers need to consider;  behavioural categories,

Analyzing Data Module 8. 2 Where are we in the Cycle? Resources Establish Need Analyze Data Interpret Data Communicate Results Use Results Plan Collect.

Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell Chapter 13 Survey Designs.

S T A T I S T I C SD E N M A R KS T A T I S T I C SD E N M A R K Test of reporting forms Ashu Conrad Statistics Denmark March 21, 2011.

Cognitive Interview Standards and Guidelines at the U. S

Introduction to Survey Research

Jennifer Crafts, Westat Audrey Kindlon, National Science Foundation,

Imran Hussain University of Management and Technology (UMT)

WHO The World Health Survey General Introduction

Concept of Test Validity

Interviews Although Social Surveys can be conducted using written questionnaires, sociologists often use the interview method as an alternative. The Interview.

DATELINE PMWG Meeting, Luxembourg, 24/25 April 2003

Marketing Research: Course 4

Presentation transcript:

Margaret Blake RESEARCH METHODS FESTIVAL: Evaluating questionnaires with cognitive testing

How do we design and administer questions in surveys in a way that will deliver valid and reliable data?

The design of survey questions is crucial You need good quality questions to produce good quality data But what makes a good survey question and how will we know if it will produce valid and reliable data?

What makes a good question? Research question and objective of survey question clearly defined Concept(s) to be included in the question are clear Research questions and concepts converted into survey questions which follow principles of good questionnaire design Question has been evaluated or tested

Evaluating questions Desk based evaluations Expert panels Standard field pilots and dress rehearsals Respondent debriefing Analysis of existing data Split ballot experiments Behaviour coding Interviewer rating Cognitive testing

The question and answer model

Cognitive testing in practice Small sample to reflect range and diversity of population Administer survey question/ instrument being tested Collect verbal reports from respondent Think aloud Probing (scripted and spontaneous) Analysis of verbal reports as qualitative data Reporting of problems with questions Recommendations of changes to questions

Youth in Europe Cross national project with testing in four countries Testing young people and parents Sample to include gender and ethnic mix Self-completion questionnaires Use of concurrent probing and think aloud Focus on problems with questions Constraints of comparability across countries Questions will be translated within each country

Variations Testing in multiple languages Test different types of instruments Diaries Card sorts Consent to data linkage Letters or other survey materials Use vignettes Test instruments in different modes Look at the question and answer process rather than testing questions

Mixed modes project Not testing questions Looking for evidence of mode effects and exploring how and why they occur Sample based on response characteristics in survey experiment, not on demographics Testing three modes in one interview Simulation of modes in face to face interview Retrospective think aloud Absence of spontaneous probing

To finish Cognitive testing offers something different There are limitations and constraints Cognitive testing can be used for more than question testing