1. Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu

2. Basic tips 1.Don’t ask what you already know 2.Avoid biased or leading questions 3.Make choice options: 1.exhaustive 2.exclusive 3.balanced 4.Avoid double-barrelled questions 5.Don’t assume knowledge of terms 6.Avoid jargon 7.Be brief 8.Be specific 9.Do/Don’t include a “neutral” option 10.Avoid complex negative constructions 11.Order questions strategically 12.Save controversial questions for last

3. Spot the flaw 1.“Rate the quality and variety of the food at Wiley Dining Court” 2.“CCO Career Services are:” ExcellentOutstandingVery goodGoodAverage 3.“Have you participated in any flipped courses during the past year?” YesNo 4.Were the clerical staff who served you in [office] courteous? 1.Yes 2.Neutral 3.No 5. What portion of the material did you master by the end of the training session? AllMostSomeVery littleNone

4. New considerations Vague quantifiers Individual response style Dunning-Kruger Effect

6. Kruger & Dunning, Journal of Personality and Social Psychology, 1999

7. What evidence of Dunning Kruger Effect looks like 2014 Purdue SERU survey

8. Vague quantifiers

10. How often have you… Had a class where the professor knew your name Communicated with an instructor by email/in person 2014 SERU Survey, Purdue data

11. Vague quantifiers “Always” almost always means always “Never” almost always means never Any other vague quantifier is... vague Vague quantifiers convey relative position only Meaning varies from one person to another Meaning varies from one question to another Grammatical opposites aren’t perceived equally

12. “Sometimes” “asking questions in class”  2 times per week “working with other students outside of class”  1 time per month “making a class presentation”  2 times per semester “participating in a community-based project”  1 time per year How Often is Often Revisited: the Meaning and Linearity of Vague Quantifiers Used on the National Survey of Student Engagement -- Laird, et al., Indiana University, 2008

13. Vague quantifiers of behavioral frequency, Marincic, 2011

14. “Stay away from using words like “often,” “usually,” “generally,” etc. Each person’s thought process is different and some people may infer a different meaning.” -- Smart Survey Design, Survey Monkey, 2011 Retrieved 3/11/2015

15. Consider using idiosyncratic terms! 2013 Purdue Student Satisfaction Survey

16. Response style

17. Moderate & Extreme Response Bursar’s drop-boxRegistering for classes 2010 Purdue Student Importance & Satisfaction Survey

18. Quotes from focus groups, Purdue, 2012-2013 “I feel like here in the US people use stronger language, like ‘Oh, this is so wonderful.’ But for me, I’m not so willing to say it.” “Americans use a lot of hyperbole: ‘It’s super, it’s wonderful!’ That is not language we use where I come from.”

19. Factors that determine acquiescence Individualism-.445* Power distance.372* Extraversion.446** Uncertainty avoidance.533* Response styles in cross-national survey research: a 26-country study, Harzing, et al, 2006

20. Response styles by nationality Turkey China

21. Some words don’t translate well “good” v. 良い / いい “normal” v. нормальный

22. Does America spend too little, about the right amount, or too much money on… General Social Survey, 2014

23. Again: Does America spend too little, about the right amount, or too much money on… Liberal respondentsConservative respondents

24. Order of choice options matters Purdue’s favorite pizza: Hot Box / Mad Mushroom / Villa@PMU / Domino’s Domino’s / Villa@PMU / Mad Mushroom / Hot Box Biasing Effects of Scale-Checking Style in Response to a Likert Scale, American Statistical Assn, Friedman, et al., 1994

25. Numbers on the scale matter, as much as the words do 012345678910 Not at allExtremely Successfulsuccessful -5-4-3-2-1012345 Not at allExtremely Successfulsuccessful Rating the Rating Scales, Journal of Marketing Management, Friedman & Amoo, 1999

26. You can get reliable results with a moderate sample size or response rate

28. After the shot went off, my training took over.

29. Resources and links: A good overall lesson in 106 slides Survey Monkey’s guide Cross–national response styles Rating the Rating Scales Vague quantifiers: How Often is Often? Vague quantifiers: Marincic dissertation More about importance/performance analysis Tex Grebner’s video (language)