What Do We Need to Consider When Designing Questionnaires and Administering Surveys? Hongfei Yang Alyse Carney Juan Manuel Balcazar

Surveys in Social Research David de Vaus

★ How to construct questionnaires? ★ The four methods of administering questionnaires.

Question Content ●Behavior Establish what people do ●Beliefs What people believe is true or false ●Knowledge Discover respondents’ knowledge of particular facts ●Attitudes Establish what people think is desirable ●Attributes Obtain information about the respondents’ characteristics

Principles of Question Design ●Reliability The questions should be answered in the same way on different occasions if given to the same person ●Validity one measures what we think it does ●Discrimination variation in the sample on the key variables ●Response rate Non-response needs to be minimised ●Same meaning for all respondents All respondents have answered the same questions ●Relevance Whether each question is necessary

Wording questions 1.Is the language simple? 2.Can the question be shortened? 3.Is the question double-barrelled? 4.Is the question leading? 5.Is the question negative? 6.Is the respondent likely to have the necessary knowledge? 7.Will the words have the same meaning for everyone? 8.Is there a prestige bias? 9.Is the question ambiguous? 10.Is the question too precise? 11.Is the frame of reference for the question sufficiently clear? 12.Does the question artificially create opinions? 13.Is personal or impersonal wording preferable? 14.Is the question wording unnecessarily detailed or objectionable? 15.Does the question have dangling alternatives? 16.Does the question contain gratuitous qualifiers? 17.Is the question a ‘head giveaway’?

Question Type Open-ended questions one for which respondents formulate their own answers. Closed questions one in which a number of alternative answers are provided from which respondents are to select one or more of the answers.

Developing Question Responses ●Exhaustiveness Ensure that the response alternatives provide a sufficient range of responses to cover all respondents. ●Exclusiveness Each question can provide one and only one answer to the question. ●Balancing Categories Where response categories can be ordered from high to low there should be the same number of response alternatives either side of what might be considered the neutral position. Example: unbalanced response alternatives: Completely approve Strongly approve Approve Neither approve nor disapprove Disapprove

Question Layout ●Answering procedures ●Contingency questions ●Instructions ●Use of space use the blank back of pages, provide a column about 2.5 centimetres wide on the right-hand side, leave sufficient space for open-ended questions, list alternative responses down, consider placing one or two question on a screen in electronic questionnaires. ●Order of questions ●Setting up for coding ●Questionnaire length

Pilot Testing: Evaluating Questions and Questionnaires

Main methods of administration ●Face-to-face interviews administer questionnaire personally to a respondent ●Telephone interviews CATI ●Postal self-administered questionnaires self-administered ●Internet surveys Web pages Combined web page and

Advantages and disadvantages of mail, face-to-face, telephone and web-based questionnaire surveys ●Response rates ●Obtaining representative sample ●Survey method effects on questionnaire design ●Quality of answers ●Implementing the survey

Questions and Answers in Attitude Surveys Howard Schuman and Stanley Presser

Question Form Versus Question Content It is useful to distinguish between certain formal features of question wording and the specific words used in any particular question. The general public often responds to more concrete phrasing of issues- “gun control”, “abortion”, etc.

Order Effects In a survey, questions are always part of a larger sequence of questions. The issue is that whether answers given to an item are influenced by the particular question or questions that precede it.

Question Selection “There are always some people whose ideas are considered bad or dangerous by other people. For instance, somebody who is against all churches and religion. If such a person wanted to make a speech in your (city/town/community) against churches and religion, should he be allowed to speak, or not?”

Question Order and Response Order “A major threat to the interpretation of any question form difference-or indeed any survey result at all- is the possible impact of preceding parts of the questionnaire.” (23) ● what looks to be a response due to question form or content may in fact be partly or entirely due to question order.

Question Order Effects Besides sampling errors, question order effects are one of the most frequently offered explanations for an unexpected or replicated survey finding. ●The context in which any question appears, or its position in a sequence of items, may conceivably influence the answers given to it.

The Frequency of Question Order Effects in an Attitude Survey ●1971 Detroit Area Study ●105 of the items did not show significant form differences, and this includes most of those near the end of the questionnaire, which means they had followed many previous variations in order.

Response Order Effects ●The order in which alternatives are read influences choice. ●Usually arise from the difficulty respondents face in attending to or keeping in mind all the alternatives presented.

Types of relations among questions and types of order effects

Part-Part Consistency Effect Example: QA. Do you think the United States should let communist newspaper reporters from other countries come in here and send back to their papers the news as they see it? QB. Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news as they see it?

Part-Whole Contrast Effects Example: QA. Are driving standards getting lower for drivers in general? QB. Are driving standards getting lower for young drivers specifically?

Part-Part Contrast Effects

Part-Whole Consistency Effects Example: General happiness: Taken altogether, how would you say things are these days: would you say that you are very happy, pretty happy, or not too happy? Marital happiness: Taking things all together, how would you describe your marriage: would you say that your marriage is very happy, pretty happy, or not too happy?

Salience effects Other types of order effects

Conclusion The art of questionnaire design involves thinking ahead about the research problem, what the concepts mean and how we will analyse the data. Form and association are very important in shaping the results of survey questions

Bibliography D. De Vaus, Surveys in Social Research, London: Routledge, H. Schuman and S. Presser, Questions and Answers in Attitude Surveys, London: Sage, 1996.