Developing Questionnaires and Questionnaire Items Keeping it Simple, Clear and Organized

Objectives Reviewing the implications of error Random sampling error Measurement error Questionnaire items that introduce error Organizing the questionnaire to improve administration of the survey Review of survey modes and types of questions

Error Components in Parameters

Addressing Error Random Sampling Error: Systematic Error Increase sample size Stratified sampling Systematic Error Composite measures Interviewer training Questionnaires that improve response rates and minimize respondent confusion.

Measurement of Meeting Expectations Previous form: Exceeding expectations Meeting expectations Almost meeting expectations Failing to meet expectations Alternative:

What would be the number of respondents from a branch necessary to determine within +/-10% (at a 95% confidence level) if a branch was significantly different on its proportion of respondents “stating” it was exceeding expectations?

Measurement of Meeting Expectations What would happen if choices were increased? Far exceeding expectations Exceeding expectations Just coming short of exceeding expectations Barely meeting expectations Falling just short of meeting expectations Failing to meet expectations No apparent attempt made to ever meet expectations

Example from 5-point scale: What is the 95% confidence interval for the population mean if the sample mean is 1.47, the sample standard deviation is 1.08, and the sample size was 36. Error or Acceptable error is +/-.35

Coding of Responses to Items and Standard Deviation Service at my FCS branch is… Getting better Getting poorer 1 2 3 1 2 3 4 1 2 3 4 5 6 7

Increasing Sensitivity By increasing the possible number of responses, you may increase the variability in responses and standard deviation. Gain “sensitivity,” But increase sample size requirements.

Questionnaire Design Determine the most important constructs or variables to be measured. What survey mode is to be used (telephone, mall interview, self-administered)? Respondent issues: Sensitivity, need to disguise questions, interest in the subject matter.

“Backward” Influence What type of analysis does the client understand? Determines the coding of variables. Coding affects the nature of the question, and time required for respondent. Determines position in the questionnaire. Position in questionnaire affects participation.

Functions of Well-Planned Questionnaires Translate research objectives into specific questions Standardized questions and response categories Foster cooperation and keep respondents motivated Serve as permanent records of the research Speed up the process of data analysis Contain information for reliability assessment

Questionnaire Items Open-ended questions pose some problem and ask the respondent to answer in his or her own words. Closed-ended give the respondent specific, limited alternative responses, and ask respondent to choose one closest to his or her viewpoint. Simple dichotomy Determinant choice (multiple choice) Frequency determination (how often do you...) Checklist, allows respondent to check multiple responses.

Likert Questionnaire Items Two components: Strongly worded statement, “My experience at this hotel was one of my best ever.” Scaled response, ranging from “strongly agree” to “strongly disagree.” Easy to create Easy for the respondent

Disguised Items: Materialism “I admire people who own expensive homes, cars, and clothes. “Some of the most important achievements in life include acquiring material possessions. “I don’t place much emphasis on the amount of material objects people own as a sign of success.* The things I own say a lot about how well I’m doing in life. I don’t pay much attention to the material objects other people own.* * Reverse coded

The Five “Shoulds” of Question Wording The Question Should... Be focused on a single issue or topic Be brief, unambiguous Be interpreted the same way by all respondents Use the respondents’ core vocabulary Be a grammatically simple sentence (if possible)

Leading Questions Suggest or imply certain answers. "Many people are using dry cleaning less because of improved wash-and-wear clothes. How do you feel wash-and-wear clothes have affected your use of dry cleaning facilities in the past four years?” “Drunken drivers continue to claim thousands of motorists’ lives each year. In the last year, has your drinking in restaurants increased or decreased?

Loaded Questions Suggests socially desirable answers or is emotionally charged. "In light of today's crisis with education in our public schools, how do you feel about the Republican Party's plan to destroy The Department of Education? “We are happy when you like programs on Channel 7. We are sad when you dislike programs on Channel 7. Write us and let us know what you think of our programming.”

Double-Barreled Question May induce bias (confusion) because it covers two issues at once. "Please indicate your agreement with the following statement: 'Wholesalers and retailers are responsible for the high cost of meat'.“ “Chuck E. Cheese’s service and food quality has been some of the best our family has experienced.”

The “Should Nots” of Question Wording: All Increase Measurement Error Assume criteria that are not obvious. “Which grocery store has best specials?” Be beyond the respondent’s ability or experience. Asking teenagers questions on household purchasing. Use a specific example to represent a general case. Including an example with a question on a broader issue. Ask the respondent to recall specifics when only generalities will be remembered Asking respondents to recall a specific price paid for a product.

Additional “Should-Nots” Requiring the respondent to guess a generalization. The question is a generalization, usually gets an affirmative response. Ask for details that cannot be related “How many brands did you consider prior to buying…?” Use words that overstate the condition “Do you believe a tuition increase will prevent the exodus of faculty from the University…” Have ambiguous wording “On your last visit to the Northwest Arkansas Mall…

Cover Letters and Questionnaire Introductions Identify the Surveyor/Sponsor of the Survey Describe the Survey’s Purpose Indicate How Prospective Respondents were Selected Request the Prospective Respondent’s Participation in the Survey/Provide incentive Screen the Prospective Respondent

Typical Question Sequence Screening Questions Warm-ups, open-ended, easy to answer Most important variables Transitions (statements & skips) Complicated, sensitive & difficult-to-answer questions Classification & demographic questions

Screening Questions “Are you the person that does the majority of the grocery shopping in your home?” “May I speak to the person who watched television last night?”

Difficult or a Warm-Up? “At what supermarkets do you regularly shop? “List the supermarkets where you regularly shop. “I’m going to read a list of supermarkets in the area, please identify if it is a store where you regularly shop.

Organization Approaches Funnel Approach—A wide to narrow approach, from general to specific questions. Work Approach—An easy-to-difficult approach, begin with brief sections and move to sections which take more time. Sections Approach—Keep the respondent on the same subject within a section. Importance—Move from most important sections to the least important. Questionnaires are a result of a combination of approaches.

Sections The questionnaire should be organized in sections. Each section should contain related questions. Use the same type of questionnaire items within a section.

Funnel Approach Applies to the organization within sections of the survey. Begin with a general question, “warm-up” respondent. Finish with a more specific question.

“Work” Approach Applies to organizations of sections within the questionnaire. Begin with shorter sections Finish with longer, more tedious sections

Importance Not all sections of the survey are of equal importance to the survey’s objectives. Do not “bury” the more important sections. If sensitive items are important, do not precede them with sections that are not critical to the survey’s objectives.

Sensitive Items Response rate decline due to respondent distrust and concern over confidentiality. Income Age Education

What is your household’s annual income to the nearest thousand? Too direct on a highly sensitive item. What percent of your household’s income is spent at grocery stores? Combined with shopping information, could provide a measure of income, but assumes respondent understands percents, and can calculate this readily (?) Is your household income above or below $50,000? Easy to answer, and can be followed up with “Above $100,000; Above 90,000; Above 70,000”

Layout of the Printed Survey Form Paper and photocopying costs, in light of other survey costs, are small. Don't crowd items on a page to make the survey appear short -- the opposite effect happens with the respondent. Keep ample empty spaces in the questionnaire booklet, it help both the interviewer and the respondent in a self-administered survey. Use vertical answer formats for individual questions. Number questions to facilitate the interviewer.

Pre-testing the Interview/Questionnaire Small subgroup of representative respondents. Identifies wording of questions and responses that may create problems. Improves interpretation of questions, reduces respondent source of error

Pre-testing the Interview/Questionnaire Can the questionnaire format be followed by the interviewer? Does the questionnaire flow naturally and conversationally? Are the questions clear and easy to understand--can the respondent answer the questions clearly? Which alternative forms of questions work best? Can the questionnaire be coded for easy data entry?