1. Developing Questionnaires and Questionnaire Items
Keeping it Simple, Clear and Organized

2. Objectives Reviewing the implications of error
Random sampling error
Measurement error
Questionnaire items that introduce error
Organizing the questionnaire to improve administration of the survey
Review of survey modes and types of questions

3. Error Components in Parameters

4. Addressing Error Random Sampling Error: Systematic Error
Increase sample size
Stratified sampling
Systematic Error
Composite measures
Interviewer training
Questionnaires that improve response rates and minimize respondent confusion.

5. Measurement of Meeting Expectations
Previous form:
Exceeding expectations
Meeting expectations
Almost meeting expectations
Failing to meet expectations
Alternative:

6. What would be the number of respondents from a branch necessary to determine within +/-10% (at a 95% confidence level) if a branch was significantly different on its proportion of respondents “stating” it was exceeding expectations?

7. Measurement of Meeting Expectations
What would happen if choices were increased?
Far exceeding expectations
Exceeding expectations
Just coming short of exceeding expectations
Barely meeting expectations
Falling just short of meeting expectations
Failing to meet expectations
No apparent attempt made to ever meet expectations

8. Example from 5-point scale:
What is the 95% confidence interval for the population mean if the sample mean is 1.47, the sample standard deviation is 1.08, and the sample size was 36.
Error or Acceptable error is +/-.35

9. Coding of Responses to Items and Standard Deviation
Service at my FCS branch is…
Getting better Getting poorer

10. Increasing Sensitivity
By increasing the possible number of responses, you may increase the variability in responses and standard deviation.
Gain “sensitivity,”
But increase sample size requirements.

11. Questionnaire Design
Determine the most important constructs or variables to be measured.
What survey mode is to be used (telephone, mall interview, self-administered)?
Respondent issues: Sensitivity, need to disguise questions, interest in the subject matter.

12. “Backward” Influence What type of analysis does the client understand?
Determines the coding of variables.
Coding affects the nature of the question, and time required for respondent.
Determines position in the questionnaire.
Position in questionnaire affects participation.

13. Functions of Well-Planned Questionnaires
Translate research objectives into specific questions
Standardized questions and response categories
Foster cooperation and keep respondents motivated
Serve as permanent records of the research
Speed up the process of data analysis
Contain information for reliability assessment

14. Questionnaire Items
Open-ended questions pose some problem and ask the respondent to answer in his or her own words.
Closed-ended give the respondent specific, limited alternative responses, and ask respondent to choose one closest to his or her viewpoint.
Simple dichotomy
Determinant choice (multiple choice)
Frequency determination (how often do you...)
Checklist, allows respondent to check multiple responses.

15. Likert Questionnaire Items
Two components:
Strongly worded statement, “My experience at this hotel was one of my best ever.”
Scaled response, ranging from “strongly agree” to “strongly disagree.”
Easy to create
Easy for the respondent

16. Disguised Items: Materialism
“I admire people who own expensive homes, cars, and clothes.
“Some of the most important achievements in life include acquiring material possessions.
“I don’t place much emphasis on the amount of material objects people own as a sign of success.*
The things I own say a lot about how well I’m doing in life.
I don’t pay much attention to the material objects other people own.*
* Reverse coded

17. The Five “Shoulds” of Question Wording
The Question Should...
Be focused on a single issue or topic
Be brief, unambiguous
Be interpreted the same way by all respondents
Use the respondents’ core vocabulary
Be a grammatically simple sentence (if possible)

18. Leading Questions Suggest or imply certain answers.
"Many people are using dry cleaning less because of improved wash-and-wear clothes. How do you feel wash-and-wear clothes have affected your use of dry cleaning facilities in the past four years?”
“Drunken drivers continue to claim thousands of motorists’ lives each year. In the last year, has your drinking in restaurants increased or decreased?

19. Loaded Questions
Suggests socially desirable answers or is emotionally charged.
"In light of today's crisis with education in our public schools, how do you feel about the Republican Party's plan to destroy The Department of Education?
“We are happy when you like programs on Channel 7. We are sad when you dislike programs on Channel 7. Write us and let us know what you think of our programming.”

20. Double-Barreled Question
May induce bias (confusion) because it covers two issues at once.
"Please indicate your agreement with the following statement: 'Wholesalers and retailers are responsible for the high cost of meat'.“
“Chuck E. Cheese’s service and food quality has been some of the best our family has experienced.”

21. The “Should Nots” of Question Wording: All Increase Measurement Error
Assume criteria that are not obvious.
“Which grocery store has best specials?”
Be beyond the respondent’s ability or experience.
Asking teenagers questions on household purchasing.
Use a specific example to represent a general case.
Including an example with a question on a broader issue.
Ask the respondent to recall specifics when only generalities will be remembered
Asking respondents to recall a specific price paid for a product.

22. Additional “Should-Nots”
Requiring the respondent to guess a generalization.
The question is a generalization, usually gets an affirmative response.
Ask for details that cannot be related
“How many brands did you consider prior to buying…?”
Use words that overstate the condition
“Do you believe a tuition increase will prevent the exodus of faculty from the University…”
Have ambiguous wording
“On your last visit to the Northwest Arkansas Mall…

23. Cover Letters and Questionnaire Introductions
Identify the Surveyor/Sponsor of the Survey
Describe the Survey’s Purpose
Indicate How Prospective Respondents were Selected
Request the Prospective Respondent’s Participation in the Survey/Provide incentive
Screen the Prospective Respondent

24. Typical Question Sequence
Screening Questions
Warm-ups, open-ended, easy to answer
Most important variables
Transitions (statements & skips)
Complicated, sensitive & difficult-to-answer questions
Classification & demographic questions

25. Screening Questions
“Are you the person that does the majority of the grocery shopping in your home?”
“May I speak to the person who watched television last night?”

26. Difficult or a Warm-Up? “At what supermarkets do you regularly shop?
“List the supermarkets where you regularly shop.
“I’m going to read a list of supermarkets in the area, please identify if it is a store where you regularly shop.

27. Organization Approaches
Funnel Approach—A wide to narrow approach, from general to specific questions.
Work Approach—An easy-to-difficult approach, begin with brief sections and move to sections which take more time.
Sections Approach—Keep the respondent on the same subject within a section.
Importance—Move from most important sections to the least important.
Questionnaires are a result of a combination of approaches.

28. Sections The questionnaire should be organized in sections.
Each section should contain related questions.
Use the same type of questionnaire items within a section.

29. Funnel Approach
Applies to the organization within sections of the survey.
Begin with a general question, “warm-up” respondent.
Finish with a more specific question.

30. “Work” Approach
Applies to organizations of sections within the questionnaire.
Begin with shorter sections
Finish with longer, more tedious sections

31. Importance
Not all sections of the survey are of equal importance to the survey’s objectives.
Do not “bury” the more important sections.
If sensitive items are important, do not precede them with sections that are not critical to the survey’s objectives.

32. Sensitive Items
Response rate decline due to respondent distrust and concern over confidentiality.
Income
Age
Education

33. What is your household’s annual income to the nearest thousand?
Too direct on a highly sensitive item.
What percent of your household’s income is spent at grocery stores?
Combined with shopping information, could provide a measure of income, but assumes respondent understands percents, and can calculate this readily (?)
Is your household income above or below $50,000?
Easy to answer, and can be followed up with “Above $100,000; Above 90,000; Above 70,000”

34. Layout of the Printed Survey Form
Paper and photocopying costs, in light of other survey costs, are small.
Don't crowd items on a page to make the survey appear short -- the opposite effect happens with the respondent.
Keep ample empty spaces in the questionnaire booklet, it help both the interviewer and the respondent in a self-administered survey.
Use vertical answer formats for individual questions.
Number questions to facilitate the interviewer.

35. Pre-testing the Interview/Questionnaire
Small subgroup of representative respondents.
Identifies wording of questions and responses that may create problems.
Improves interpretation of questions, reduces respondent source of error

36. Pre-testing the Interview/Questionnaire
Can the questionnaire format be followed by the interviewer?
Does the questionnaire flow naturally and conversationally?
Are the questions clear and easy to understand--can the respondent answer the questions clearly?
Which alternative forms of questions work best?
Can the questionnaire be coded for easy data entry?