Surveys Alfred Kobsa University of California, Irvine

When to chose a survey Scope of problem very clear Questions can be clearly formulated at a concrete level Complex/holistic answers not needed. Checking pre- formulated answers is sufficient (plus occasionally one or two free-text sentences). Mailing lists with current addresses available Specific location not required Answers from many users needed Answers needed quickly Broad geographic distribution of respondents needed Anonymity required

Steps Identify the objectives of the study (in writing) Select the target audience Select the form of announcement and response collection ( , physical mail, web [ ☛ survey websites], in person, phone) Decide how to analyze the data (software, statistical tests if any) Brainstorm questions (closed questions, open questions) Formulate questions (mind double-barreled, double negatives, leading or loaded questions, and questions with self-image, acquiescence and social desirability bias) Use existing surveys (e.g. System Usability Scale SUS ) Consider capabilities of survey site Reduce question set Pilot test and revise your questions!

Problems to consider Non-representative sample population ( ☛ compare demographics of respondents with demographics of names in mailing list and normalize answers accordingly) Low response rate ( ☛ pre-announcement, personalized cover letter, reward, short survey, few open-ended questions, self- addressed envelope w/ stamp, multi-mode reminders) Self-selection bias of respondents based on topic ( ☛ hide your study goal among more general questions) Attention, honesty ( ☛ reverse-code questions, check consistency, discard data) Self-incrimination ( ☛ emphasize“harmlessness”, impunity, anonymity) Social desirability bias, self-image bias ( ☛ phrase questions carefully, use several questions for same concept, go for facts and not habits, attitudes or intentions) Sensitive questions ( ☛ explain their purpose, do not make answers mandatory, offer broad answer bins)

Response format Closed-ended questions Single-choice or multiple-choice (include “other” + “explain”) Ratings (“on a scale from 1 to 5, how would you rate…”) Likert* scale: [1 … n] or [-n -n+1…. n-1 n] ( typically n = 5, 6 or 7) with mid-point or without (forces respondents to take a stance) with verbal “anchors” for first and last value only, or for all/some values including first and last** Ranking scales (e.g., sorting by priority) Open-ended questions Should be kept to a minimum and optional since they may cause users not to fill in or abandon a questionnaire if users answer them, they only provide brief answers answers are hard to evaluate *) pronounced like “ike” in the US, but like “lickert” elsewhere (also by Mr Likert himself and his son) **) First and last value anchors should be “extreme”