Collective Wisdom Designing Surveys for Use in Assessment R. Ternes, October 2011

Defining the Problem The most important consideration when creating a good survey is to identify and define your problem/question/goal. ◦ Vague problems lead to vague surveys which lead to vague answers. Chunk the problem ◦ Break broad categories into manageable and measurable parts.

Examples Too Broad ◦ “What is our student learning environment like?” Break it down. ◦ How many hours do students study? ◦ How many students study in groups? ◦ How many papers do students write? ◦ How many students attend office hours? ◦ etc. etc.

Sampling and Generalizability Sampling is linked to your goal. ◦ Tackle sampling after defining your goal. You must sample from the population that you want to make generalizations about. ◦ If you want to generalize to all your students, you must sample from all your students.  Sampling English majors will tell you only about English majors.

Convenience Sampling An ideal survey uses random sampling drawn from the population of interest. Often, we have to make compromises to the way we sample. ◦ We might sample our first-year students by surveying students in their first year writing course. Understand the limitations of your choice!

Understanding Limitations Consider the CLA, CAAP, MAAP etc. as a Gen Ed student learning measure. ◦ For seniors, sample from everyone. ◦ For freshmen, sample from students that enroll in your first year writing program (because it’s convenient).  If you do so, do you exclude your AP students?  Aren’t these some of your best students?

Writing Good Survey Questions Good questions are: ◦ Short ◦ Simple ◦ Clear ◦ And they obviously relate to the goal or problem!

Writing Good Survey Questions Binary or Likert scales are good standards ◦ 5, 7, or 10 point scales are standard. ◦ Right Hand Side = high numbers/pos. phrases ◦ Use n/a only if you need to.  Always make it a separate option.

Writing Good Survey Questions Use standard phrasing. ◦ Strongly agree, agree, disagree, etc. ◦ Don’t use words like ‘terrific’ or ‘terrible’. ◦ Keep the language simple and plain. ◦ Try to make the scale ‘balanced’.  One option for a negative weighting and 3 options for a positive rating ‘anchors’ the results towards the positive side.

Standard Format The drinking age should be lowered from 21 to 18. ◦ Strongly Agree ◦ Agree ◦ Neutral ◦ Disagree ◦ Strongly Disagree This format naturally avoids several common pitfalls in survey design.

Use Non-Standard Formats Sparingly Try to avoid non-standard question formats like: ◦ ‘rank these in order of preference’ ◦ ‘select all that apply’ Non-standard formats ◦ are more difficult to analyze statistically, ◦ Are often harder to report, ◦ and they can confuse participants. ◦ Sometimes they can be very useful, but these questions are easy to do poorly.

Bad Question Overview Leading Questions Manipulating the Question Order Double-Barrelled ‘Either-or’ Fallacies Lack of Variability Lengthy/Complex Questions

Leading Questions Leading questions steer respondents towards a particular answer. ◦ Bad: “Do you think tuition is too high?” ◦ Better: “How would you best classify your school’s tuition cost?”  Much higher than other comparable schools  Higher than other comparable schools  About the same as other comparable schools  Lower than other comparable schools  Much lower than other comparable schools

Question Order Question order is important! If you are polling opinions on government involvement in health care: ◦ You are likely to illicit more negative responses if you first ask a dozen questions about government spending. Put controversial questions last, if possible.

Double-Barreled Questions Do not roll two questions into one. “Do you like the flavor and texture of the cereal?” ◦ If you get a ‘no’ response:  Is it the flavor they did not like?  Is it the texture they did not like?  Is it both?

Turn Either-Or Binaries into Likerts Resist the temptation to create binary variables out of everything. ◦ Are you a Democrat or a Republican?  Such a question ignores people that are Independent, Libertarian, Green Party, Tea Party, etc. ◦ Researchers can lose valuable information by dichotomizing continuous variables. Often binary questions can be easily turned into Likert type scales.

Lack of Variability Design questions so that they give you information! ◦ If everyone answers in the same way, then you haven’t gained much information. Bad ◦ Do you want more parking on campus? Better ◦ Do you think the university should create a $5 student fee to build more parking spots?

Avoid Lengthy Questions and Double Negatives Don’t use double negatives. Simplify, Simplify, Simplify. ◦ Change: “At what convenience store were you or a member of your family most likely to purchase their gasoline from in the past year?” ◦ To: “What gas station do you usually buy gasoline from?”