DESIGNING GOOD SURVEYS Laura P. Naumann Assistant Professor of Psychology Nevada State College

Survey Topic Brainstorming ■What do you hope to learn from your participants? –List topics you want to measure ■Who do you need to survey? –List groups of people –e.g., faculty; staff; students; married people; over 40 ■Who do you plan to share the findings with? –Internal audience ■Yourself ■Your team/unit/NSC –External audience ■Professional conference ■Research journal or publication

Pre-Planning ■What do you want to learn? –Make a list of all possible topics you need to measure –Organize the topics in a meaningful order ■Who will take your survey? –Can all participants respond to all the same questions? ■If not, you may need to make two different surveys or use SKIP LOGIC to hide questions. ■Who are you sharing your findings with? –Internal audience –External audience ■Need IRB approval ■Consider sophistication of statistical analyses

Measuring Your Topics Things to Consider: ■Will your participants feel comfortable answering questions about the topics you’ve selected? –Consider sensitive issues (income; health; relationships) –Consider your demographic questions (too intrusive?) –Take precautions to preserve anonymity. ■What type of data is acceptable to you? –Frequency counts or Percentages (# or % of people) –Means and Standard Deviations (average score) –Difference between groups (who’s higher or lower?) –Relationship between variables (correlations)

TYPES OF QUESTIONS

Riddle me this… What’s the difference between these two options? How will you summarize your results?

Forced-choice vs. Checkbox Options Results = # or % of people from a given sample who enjoy each flavor; sum of total equals 100% Results = # or % of people who like each flavor; sum of totals exceed 100% Mutually exclusive choices Can make many choices

Types of Questions What’s the difference between these question types? How will you summarize your results?

Categorical Variables ■Categorical data results from questions that use mutually exclusive answer options (i.e., categories). ■Can only summarize data using frequency counts or percentages.

Continuous Variables ■Continuous data results from questions that provide a numerical range of responses (e.g., 5-pt scale; enter number as an answer). ■Best summarized using means and standard deviations. ■Can also be summarized with median score; frequency at each level. ■Can be used to make comparisons between groups or correlations.

Categorical vs. Continuous Scales CATEGORICALCONTINUOUS Frequency counts % in each group Mean, median, mode Range, standard deviation How would you turn this question into a continuous measure of time spent on homework?

Review

Tips for Programming CONTINUOUS

CHOOSING SCALE ANCHORS See Handout on Common Anchors!

Choose Anchors Wisely! QUESTION: How interesting was the workshop? These answer options cannot be ordered numerically. They would give you categorical data. These answer options can be ordered numerically (i.e., continuous). Will provide means/standard deviations. 1 = not at all 2 = slightly 3 = moderately 4 = very 5 = extremely QUESTION: How interesting was the workshop?

Match anchors to question! What is the researcher trying to measure with this question? How can we re-word the question to better measure the “usefulness of the workshop”? Match question to anchors: 1 = Not at all 2 = A little 3 = Somewhat 4 = Very 5 = Extremely

Common Response Anchors ■Agreement: Strongly disagree to strongly agree. ■Frequency: Never to often. ■Quality: Very bad to very good. ■Likelihood: Never to definitely. ■Importance: Not at all important to very important. Higher numbers (5) should always indicate more (not less) agreement or satisfaction.

5- & 7-point Examples

QUESTION WORDING

Write clear, understandable statements! PROBLEMATIC: Problems with the wording? Problems with scale anchors? What does the researcher want to measure? How well the health care professional knows about organ donation in order to make recommendations to patients… How familiar are you with: Continuous scale (means)

Watch your wording! Biased, leading question PROBLEM Long, complicated Provide scenario/details separate from questions; Ask about each item separately SOLUTION Stay objective; remove leading language Match anchors (poor, fair, good, very good) Vague/abstract concept Clarify with examples Ask about multiple examples to assess important distinctions Combines two ideas into one question Separate ideas into two different questions

Review

Final Thoughts! ■Always consider the type of data you need to conduct the statistical analyses you want BEFORE writing your survey! ■Provide meaningful scale anchors – take the survey yourself to ensure each prompt + anchor makes sense. ■Can all of your participants reasonably answer all of the questions? –e.g., If asking relationship questions, how should single people respond? ■Consider the order of your survey – will seeing one section before another skew possibly answers? –Group like-minded questions together –Intermix questions using the same response scale ■Consider the length of your survey – only include questions that are absolutely necessary. –Provide a status bar; ability to save; move forward/back

Online Resources ■Research Methods Knowledge Base – ■My Market Research Methods – design-best-practices/ design-best-practices/ ■List of Scale Anchor Options – institutes/tourism/documents/sample-scales.pdfhttps:// institutes/tourism/documents/sample-scales.pdf