Questionnaire Design Clinic

Slides:



Advertisements
Similar presentations
Jack Jedwab Association for Canadian Studies September 27 th, 2008 Canadian Post Olympic Survey.
Advertisements

WPA-WHO Global Survey of Psychiatrists' Attitudes Towards Mental Disorders Classification Results for the Spanish Society of Psychiatry.
1 Inducements–Call Blocking. Aware of the Service?
Using Matrices in Real Life
Copyright © 2003 Pearson Education, Inc. Slide 1 Computer Systems Organization & Architecture Chapters 8-12 John D. Carpinelli.
McGraw-Hill/Irwin Copyright © 2013 by The McGraw-Hill Companies, Inc. All rights reserved. Extended Learning Module D (Office 2007 Version) Decision Analysis.
1 Copyright © 2013 Elsevier Inc. All rights reserved. Chapter 3 CPUs.
OvidSP Flexible. Innovative. Precise. Introducing OvidSP Resources.
Alignment and Quality 2/8/2014 Alignment and Quality 1 What is it? What do I need to look for?
Slide 1 FastFacts Feature Presentation August 28, 2008 We are using audio during this session, so please dial in to our conference line… Phone number:
Slide 1 FastFacts Feature Presentation August 12, 2010 We are using audio during this session, so please dial in to our conference line… Phone number:
Slide 1 FastFacts Feature Presentation September 7, 2010 We are using audio during this session, so please dial in to our conference line… Phone number:
Slide 1 FastFacts Feature Presentation October 15, 2013 To dial in, use this phone number and participant code… Phone number: Participant.
UNITED NATIONS Shipment Details Report – January 2006.
DIVERSE COMMUNITIES, COMMON CONCERNS: ASSESSING HEALTH CARE QUALITY FOR MINORITY AMERICANS FINDINGS FROM THE COMMONWEALTH FUND 2001 HEALTH CARE QUALITY.
1 RA I Sub-Regional Training Seminar on CLIMAT&CLIMAT TEMP Reporting Casablanca, Morocco, 20 – 22 December 2005 Status of observing programmes in RA I.
DRDP Measure Slides by Domain
Preschool Learning Foundations and Curriculum Framework, Volume 2
Building: Knowledge, Security, Confidence Pay Yourself First FDIC Money Smart for Young Adults.
Conversion Problems 3.3.
Prepared by: Workforce Enterprise Services For: The Illinois Department of Commerce and Economic Opportunity Bureau of Workforce Development ENTRY OF EMPLOYER.
Create an Application Title 1A - Adult Chapter 3.
1 Career Pathways for All Students PreK-14 2 Compiled by Sue Updegraff Keystone AEA Information from –Iowa Career Pathways –Iowa School-to-Work –Iowa.
FACTORING ax2 + bx + c Think “unfoil” Work down, Show all steps.
SADC Course in Statistics Session 4 & 5 Producing Good Tables.
1 © 2009 University of Wisconsin-Extension, Cooperative Extension, Program Development and Evaluation Response Rate in Surveys Key resource: Dillman, D.A.,
Copyright 2009 National Student Loan Program Creating Surveys that Work!
Copyright ©2010 Pearson Education, Inc. publishing as Prentice Hall
1 SESSION 5- RECORDING AND REPORTING IN GRADES R-12 Computer Applications Technology Information Technology.
Purpose To create the distribution of contents from the unit three.
1. 2 Evaluation Report A preliminary report to the faculty and administrators of the online distance learning program in the Department of Educational.
User Friendly Price Book Maintenance A Family of Enhancements For iSeries 400 DMAS from Copyright I/O International, 2006, 2007, 2008, 2010 Skip Intro.
The Frequency Table or Frequency Distribution Table
Chi-Square and Analysis of Variance (ANOVA)
Results from a Mobile Finance Survey. 2 2 Second survey sponsored by CheckFree with fieldwork in April 2008; First survey completed in March ,007.
Success Planner PREPARE FOR EXAMINATIONS Student Wall Planner and Study Guide.
Student Focus Groups UNIVERSITY OF KENTUCKY COLLEGE OF PHARMACY 1 Formative Evaluation Using Student Focus Groups Heidi M. Anderson, Ph.D. University of.
Developing a Questionnaire
Promoting Regulatory Excellence Self Assessment & Physiotherapy: the Ontario Model Jan Robinson, Registrar & CEO, College of Physiotherapists of Ontario.
 Copyright I/O International, 2013 Visit us at: A Feature Within from Item Class User Friendly Maintenance  Copyright.
Data Management Seminar, 8-11th July 2008, Hamburg Survey System – Overview & Changes from the Field Trial.
Developing a Global Vision Through Marketing Research
Functional Areas & Positions
Model and Relationships 6 M 1 M M M M M M M M M M M M M M M M
Business documents and correspondence. Housekeeping › mobile phones › break times › toilets › emergencies © smallprint 2.
What is a Survey? A scientific social research method that involves
Student Survey
Chapter 12 Working with Forms Principles of Web Design, 4 th Edition.
Essential Cell Biology
1 Phase III: Planning Action Developing Improvement Plans.
Intracellular Compartments and Transport
PSSA Preparation.
Essential Cell Biology
Energy Generation in Mitochondria and Chlorplasts
By Hui Bian Office for Faculty Excellence Spring
Profile. 1.Open an Internet web browser and type into the web browser address bar. 2.You will see a web page similar to the one on.
The Framework for Teaching Charlotte Danielson 4c: Communicating with Families 1 6/12/201 3.
Data, Now What? Skills for Analyzing and Interpreting Data
1 Sources:  SusanTurner - Napier University  C. Robson, Real World Research, Blackwell, 1993  Steve Collesano: Director, Corporate Research and Development.
Design Questionnaire Gary Ma, IC. Sources of information Josie Csete, Seminar on “Tips of creating and conducting survey”, EDU HKPU, April 15, POSH,
Introduction to Questionnaire Design
Building a Better Survey Katie Busby, PhD Director of Student Affairs Assessment & Planning University of Alabama Keston Fulcher, PhD Director of Assessment,
Data Collection Methods
Chapter 12 Survey Research.
QUESTIONNAIRE DESIGN: FORMAT Lu Ann Aday, Ph.D. The University of Texas School of Public Health.
Introduction to Questionnaire Design
DESIGNING GOOD SURVEYS Laura P. Naumann Assistant Professor of Psychology Nevada State College.
Introduction to Survey Design
Presentation transcript:

Questionnaire Design Clinic Spring 2010 Seminar Series Survey Research Laboratory University of Illinois www.srl.uic.edu

Cognitive Steps in Answering Questions Understand question. Search memory for information. Integrate information into summary judgment. Translate judgment onto response alternatives.

Survey Intro/Cover Letter Introduction should indicate: who is conducting the survey the topics to be covered in the survey an assurance of confidentiality any IRB stipulations whether you offer how long it will take depends on mode, topic, population

Survey Intro/Cover Letter - example The  goal of this survey is to understand the patterns of interactions between the groups and individuals who will be attending ….. The results of this survey will be used in part for University of Illinois at Chicago research purposes. Ultimately, we hope to provide a visual representation of the social network structure of various interested parties in the ____________ region. By completing this survey, you are agreeing to participate in the research.

Cover letter - example The Department of _________at the University of Illinois at Chicago, is conducting to understand the patterns of interactions between the groups and individuals ….… The research team is headed by… You are receiving this questionnaire because you are . . . It should take you about 10 minutes to complete the questionnaire. Please respond by August 1. Participation in this survey is voluntary, and you may choose to stop answering questions at any time. Your choice to participate or not will have no effect on your course grade or on your relation with the University of Illinois. Your response is anonymous as we will not be asking for any personally identifying information – please do not write your name anywhere on this questionnaire. We plan to use the results of this survey in part for University of Illinois at Chicago research purposes. Ultimately, we hope to provide a visual representation of the social network structure of various interested parties in the Calumet region.……. Only aggregated results will be presented. Completing this questionnaire indicates your consent to participate in the survey. Please contact ________ if you have any questions. If you have any questions about your rights as a research participant, please contact the IRB (OPRS) at…. Thank you for your participation! Names Affiliation Email / phone number

Ordering the Questions First questions should be directly related to the topic as described in the introduction or advance/cover letter Put demographic questions at the end Leave objectionable questions (e.g., income) for the end

Characteristics of a good question One that yields a truthful, accurate answer One that asks for one answer on one dimension One that accommodates all possible contingencies of response One that uses specific, simple language One that produces variability in response One that minimizes social desirability One that is pretested

Acquiescence - example Would you be interested in learning about courses from the past experiences of other students who have taken them? Yes No Do you feel ready to apply what you have learned today?

Acquiescence - example Administrators (e.g., principal, assistant principal, dean of students ) at this school ……………… Collaborate with faculty and staff to make the school run effectively. Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree

Acquiescence Form A (agree I’s version): Individuals are more to blame than social conditions for crime and lawlessness in this country. Form A (Agree I’s) Agree (I): 59.6% Disagree (SC): 40.4 100 (473) Form B (agree SCs version): Social conditions are more to blame than individuals for crime and lawlessness in this country. Form B (Agree SCs) Agree (SC): 56.8% Disagree (I): 43.2 100 (472)

Acquiescence - example Do you feel ready to apply what you have learned today? Yes No Do you feel ready to apply what you have learned today, or do you not feel ready to do that? Feel ready Do not feel ready

Acquiescence - example Would you be interested in learning about courses from the past experiences of other students who have taken them? Yes No How interested are you in learning about courses from the past experiences of other students who have taken them? Not at all interested, Slightly interested, Moderately interested, Very interested, Extremely interested

Characteristics of a good question - example Administrators (e.g., principal, assistant principal, dean of students ) at this school collaborate with faculty and staff to make the school run effectively. Strongly Agree, Agree, Neutral, Disagree, Strongly Disagree   How well do administrators (e.g., principal, assistant principal, dean of students ) at this school collaborate with faculty and staff to make the school run effectively?   Not well at all, Slightly well, Moderately well, Very well, Extremely well

Characteristics of a good question One that yields a truthful, accurate answer One that asks for one answer on one dimension One that uses specific, simple language One that accommodates all possible contingencies of response One that produces variability in response One that minimizes social desirability One that is pretested

Characteristics of a good question - example The instructor was able to answer questions clearly and concisely in this course. Participants materials (high quality; easy to follow; helpful, etc)

Characteristics of a good question One that yields a truthful, accurate answer One that asks for one answer on one dimension One that uses specific, simple language One that accommodates all possible contingencies of response One that produces variability in response One that minimizes social desirability One that is pretested

Characteristics of a good question - example Indicate how the amount of use of the following types of classroom assessment have changed in your classroom Assessment during instruction to determine: degree of comprehension on part of an individual student…..

Characteristics of a good question - example Has your use of the following types of assessments increased, decreased or stayed the same? Assessment during instruction to find out how well a student has understood the material…..

Characteristics of a good question One that yields a truthful, accurate answer One that asks for one answer on one dimension One that uses specific, simple language One that accommodates all possible contingencies of response One that produces variability in response One that minimizes social desirability One that is pretested

Characteristics of a good question - example How often do you go talk with your academic advisor in the Department of Communication? Multiple times a semester Once a semester Once a school year For emergencies only How often do you get in touch with your academic advisor in the Department of Communication? Never Once a school year or less often Once a semester Multiple times a semester or more often In emergencies only

Characteristics of a good question One that yields a truthful, accurate answer One that asks for one answer on one dimension One that uses specific, simple language One that accommodates all possible contingencies of response One that produces variability in response One that minimizes social desirability One that is pretested

Pretesting Preferable to test the questionnaires with people like those in your main study population Test in same mode to be used for the study Consider cognitive pretesting

Other characteristics of a good question Specify who, what, when, where and how. Over what period of time? What’s included? Provide cues

Other characteristics of a good question - example Does this table of contents cover recent developments? Does this table of contents cover recent research / topics in this field, or does it not? How many people live in your household? Please include yourself in the count, and also those who people who live with you but are away temporarily. Don’t forget to count any babies and children. How many years have you been enrolled in a college or University? Not counting summers, for how many years have you been enrolled as a student in a college or university? Please count each year that you were enrolled even if these are not consecutive years.

Open vs. Closed Questions Closed questions are usually better Easier for the respondent Less coding later Better to have respondent do categorizing Categories help define the question

Open vs. Closed Questions - example What grade level(s) have you worked with in the past 12 months? (e.g., 6th, 7th, 8th) _________________ Which grade level(s) have you worked with in the past 12 months? (please select all that apply). Grade 6 Grade 7 Grade 8 How often do you send text messages? _________________ How often do you send text messages? You can answer in any of the three different ways below, whichever one would be most accurate for you. You can tell us how many times per DAY you send text messages, or how many times per WEEK you send text messages, or how many times per MONTH you send text messages.

Disadvantages of Closed Questions Categories may be leading to respondents May make it too easy to answer without thinking May limit spontaneity Not best when asking for frequency of sensitive behaviors there are numerous possible responses role of initial question development and pretesting

Response Options Response categories should be consistent with the question Categories should typically be exhaustive, including every possible answer Categories should be mutually exclusive “Don’t know” category usually not appropriate knowledge questions

Response Options - example Did the facilitator have a thorough understanding of the subject matter? Very Poor, Needs Improvement, Okay, Good, Excellent How good or bad was the facilitator’s understanding of the subject matter? Extremely good, Somewhat good, Neither good nor bad, Somewhat bad , Extremely bad

Response Options - example Please indicate below which of the following organizations you interact with on a regular basis. Please check all that apply. Organization A I regularly exchange ideas or advice with members of this organization. I regularly collaborate on projects with members of this organization. I have personal friends in this organization. I provide funding or receive funding from this organization. How often do you exchange ideas or advice with members of each of the following organizations? Organization A: Never, Rarely, Sometimes, Very Often, Extremely Often Organization B: Never, Rarely, Sometimes, Very Often, Extremely Often Do you have at least one personal friend in each of the following organizations, or do you not? Organization A: Have, Do not have Organization B: Have, Do not have

Response Options 5-7 response options General guidelines number of scale points (sensitivity vs. meaning) labeling of scale points

Response Option Labels - example How would you rate the instructor’s ability to explain the material in this course? Extremely poor, Somewhat poor, Fair, Somewhat good, Extremely good (or Excellent)

Physical Format Checklist Number all questions sequentially Use large, clear type; don’t crowd ‘White space:’ Place more blank space between questions than between subcomponents of questions List answer categories vertically instead of horizontally Avoid double/triple ‘banking’ of response choices Be consistent with direction of response categories Be consistent with placement of response categories

Physical Format Checklist (continued) Don’t split questions across pages. If necessary (e.g., question requires 1.5 pages), restate question and response categories on next page Put special instructions on questionnaire as needed, next to question Distinguish directions from questions – be consistent

Key References To Get Started Dillman, Don. Mail and Internet Surveys: The Tailored Design Method. New York: John, Wiley & Sons, Inc, 1999. Bradburn, N, Sudman, S. and Wansink, B. Asking Questions: The Definitive Guide to Questionnaire design – for Market Research, Political Polls, and Social and Health Questionnaires. San Francisco: Jossey Bass, 2004. See also past issues of Public Opinion Quarterly

Other References Schuman, H., & Presser, S. (1981). Questions and answers in attitude surveys. New York: Academic Press. Krosnick, J.A., Holbrook, A. L., Berent, M. K., Carson, R. T., Hanemann, W. M., Kopp, R. J., Mitchell, R. C., et al. (2002). The impact of "No Opinion" response options on data quality: Non-attitude reduction or an invitation to satisfice? Public Opinion Quarterly, 66, 371–403. Saris, W. E., Krosnick, J. A., & Schaeffer, E. M. (2005). Comparing questions with agree/disagree response options to questions with construct-specific response options.Unpublished manuscript, Political, Social, Cultural Sciences, University of Amsterdam.