Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521

Basic tips 1.Don’t ask what you already know 2.Avoid biased or leading questions 3.Make choice options: 1.exhaustive 2.exclusive 3.balanced 4.Avoid double-barrelled questions 5.Don’t assume knowledge of terms 6.Avoid jargon 7.Be brief 8.Be specific 9.Do/Don’t include a “neutral” option 10.Avoid complex negative constructions 11.Order questions strategically 12.Save controversial questions for last

Spot the flaw 1.“Rate the quality and variety of the food at Wiley Dining Court” 2.“CCO Career Services are:” ExcellentOutstandingVery goodGoodAverage 3.“Have you participated in any flipped courses during the past year?” YesNo 4.Were the clerical staff who served you in [office] courteous? 1.Yes 2.Neutral 3.No 5. What portion of the material did you master by the end of the training session? AllMostSomeVery littleNone

New considerations Vague quantifiers Individual response style Dunning-Kruger Effect

Kruger & Dunning, Journal of Personality and Social Psychology, 1999

What evidence of Dunning Kruger Effect looks like 2014 Purdue SERU survey

Vague quantifiers

How often have you… Had a class where the professor knew your name Communicated with an instructor by /in person 2014 SERU Survey, Purdue data

Vague quantifiers “Always” almost always means always “Never” almost always means never Any other vague quantifier is... vague Vague quantifiers convey relative position only Meaning varies from one person to another Meaning varies from one question to another Grammatical opposites aren’t perceived equally

“Sometimes” “asking questions in class”  2 times per week “working with other students outside of class”  1 time per month “making a class presentation”  2 times per semester “participating in a community-based project”  1 time per year How Often is Often Revisited: the Meaning and Linearity of Vague Quantifiers Used on the National Survey of Student Engagement -- Laird, et al., Indiana University, 2008

Vague quantifiers of behavioral frequency, Marincic, 2011

“Stay away from using words like “often,” “usually,” “generally,” etc. Each person’s thought process is different and some people may infer a different meaning.” -- Smart Survey Design, Survey Monkey, 2011 Retrieved 3/11/2015

Consider using idiosyncratic terms! 2013 Purdue Student Satisfaction Survey

Response style

Moderate & Extreme Response Bursar’s drop-boxRegistering for classes 2010 Purdue Student Importance & Satisfaction Survey

Quotes from focus groups, Purdue, “I feel like here in the US people use stronger language, like ‘Oh, this is so wonderful.’ But for me, I’m not so willing to say it.” “Americans use a lot of hyperbole: ‘It’s super, it’s wonderful!’ That is not language we use where I come from.”

Factors that determine acquiescence Individualism-.445* Power distance.372* Extraversion.446** Uncertainty avoidance.533* Response styles in cross-national survey research: a 26-country study, Harzing, et al, 2006

Response styles by nationality Turkey China

Some words don’t translate well “good” v. 良い / いい “normal” v. нормальный

Does America spend too little, about the right amount, or too much money on… General Social Survey, 2014

Again: Does America spend too little, about the right amount, or too much money on… Liberal respondentsConservative respondents

Order of choice options matters Purdue’s favorite pizza: Hot Box / Mad Mushroom / / Domino’s Domino’s / / Mad Mushroom / Hot Box Biasing Effects of Scale-Checking Style in Response to a Likert Scale, American Statistical Assn, Friedman, et al., 1994

Numbers on the scale matter, as much as the words do Not at allExtremely Successfulsuccessful Not at allExtremely Successfulsuccessful Rating the Rating Scales, Journal of Marketing Management, Friedman & Amoo, 1999

You can get reliable results with a moderate sample size or response rate

After the shot went off, my training took over.

Resources and links: A good overall lesson in 106 slides Survey Monkey’s guide Cross–national response styles Rating the Rating Scales Vague quantifiers: How Often is Often? Vague quantifiers: Marincic dissertation More about importance/performance analysis Tex Grebner’s video (language)