HOW TO WRITE GOOD SURVEY QUESTIONS Amy Moffat, PhD

Characteristics of a Survey Questionnaires Large number of respondents (if small number, better to do interviews) Either use probability or nonprobability sampling Systematic procedures are used to ask predetermined questions and record answers Answers are numerically coded and analyzed with aid of statistical software Probability sample = randomized Nonprobability => can NOT infer about the larger population

How to Ask Survey Questions CLARITY Questions should mean the same thing to every respondent 6th grade reading level Short questions are best CONSISTENCY Keep directions the same, or as similar as possible Keep answers to positive/negative questions in same question (unless there is a good reason to mix it up) Cluster similar types of items together

How to Ask Survey Questions OPEN-ENDED AND CLOSE-ENDED QUESTIONS Close-ended questions ensure uniformity, can be easier for the subject to answer, and can be easier to analyze Open-ended questions allow for more subtle responses and perhaps more precise measurement

ISSUE: Double-barreled Avoid double-barreled questions, which are questions that “force” two sentiments to coexist Example: Please indicate how much you agree or disagree with each of the following statements about the childcare program. Incorrect: I feel welcomed by staff and other youth at the center Correction: I feel welcomed by staff at the center I feel welcomed by other youth at the center

ISSUE: Biased / Leading questions Avoid biased questions Social desirability is a common problem Avoid associating an attitude or a position with a prestigious person or agency Include both positive and negative choices in the question Be aware of “loaded” terms Be aware of “acquiescence” – which is when a respondent “gives up” and provides similar responses to similarly formatted question

ISSUE: Biased / Leading questions Incorrect: Community organizing is hard. Do leadership trainings help you feel prepared for community organizing? Much more prepared Somewhat more prepared Slightly more prepared Not more prepared Correction: The leadership trainings prepare me for community organizing. Strongly agree Agree Disagree Strongly disagree

ISSUE: Double negatives Avoid questions with double negatives It could be a negative statement containing two negative elements (didn't say nothing ). Or it could be a positive statement in which two negative elements are used to produce the positive force, usually for some particular rhetorical effect (there is not nothing to worry about! ). Incorrect Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened? Very possible Possible Impossible Very impossible Correct Do you doubt that the Holocaust actually happened or not?

ISSUE: Accessible Information Questions in a survey need to be answerable Avoid questions respondents may not know the answer to One option is to use a filter question SKIP patterns are very helpful Computerized surveys and questionnaires can be programmed so that respondents never see that questions have been skipped based on earlier responses.

Example #1 More people have attended the movie, Gone with the Wind, than any other motion picture produced this century. Have you seen this movie? Yes No Problem? Biased / Leading Solution? Have you seen the movie Gone with the Wind?

Example #2 In your opinion, how would you rate the speed and accuracy of your work? Excellent Good Fair Poor Problem? Double-barreled Solution? In your opinion, how would you rate the speed of your work? In your opinion, how would you rate the accuracy of your work?

Example #3 Did you first hear about the bombing: Problem? Solution? ______ from a friend or relative ______ from a newspaper ______ from the television or radio or other electronic media ______ from your spouse ______ at work Problem? Categories are not mutually exclusive Solution? ____ from a friend ____ from a relative ____ from a media source

Example #4 How do you feel about the following statement? We should not reduce military spending. Strongly agree Agree Disagree Strongly disagree Problem? Double Negative Solution? How do you feel about the following statement? We should reduce military spending. Strongly agree Agree Disagree Strongly disagree

Example #5 Higher than single-family density is acceptable in order to make housing affordable. Strongly Agree Agree Undecided Disagree Strongly Disagree Problem? Ambiguous Solution? In order to make housing affordable, is it acceptable to build housing whose buildings hold more than one family?

Example #6 People grow up in all different types of families. What type of family did you grow up in? ______Mom as single parent ______Dad as a single parent ______Both Mom and Dad Problem? Categories are not exhaustive Solution? People grow up in all types of families. What type of family did you grow up in? ____two biological parents ____one biological parent & a step parent ____ mom as a single parent ____ dad as a single parent ____ with a relative other than my biological parents ____ in foster care ____ with adoptive parent(s)

Critique Real Life Example