1. Surveys Julie Demuth (With input from Jeff Lazo, Mary Hayden, other resources … and hopefully you!) Summer WAS*IS, July 14, 2007

2. Outline What is survey research? To use surveys or not: Some factors and information to consider Survey research components  emphasis on design Remember, this is just a brief introduction!

3. What is survey research? Surveys –Self-administered questionnaires (written) –Face-to-face interview, telephone interview, etc. (oral) –Cross-sectional design More than 1 case, single point in time, analyzed to detect patterns of association

4. To use surveys or not? Objectives –What information do I need? –What will I do with the information? –How will I analyze the information? –How and to whom will I present the results?

5. Advantages to surveys Gathering information –Efficient for collecting large amount of information about a large population –Flexible to collect wide range of information (e.g., attitudes, values, beliefs, past behaviors) Questions –Subjects gets same questions –Researcher can ask more complex questions –No response effect (willing to divulge more info w/o face- to-face contact; less likely to try to impress interviewer) Administering –Relatively easy and inexpensive to administer –Can be computer-based

6. Disadvantages to surveys Gathering information –Questions must be general enough to be appropriate for all respondents, possibly missing important context –Useless with non-literate or illiterate populations, only English-speaking populations, or hard-to-reach populations Questions –Subjects’ motivation, memory, and ability to respond –Not appropriate for studying complex social phenomena –Subjects may not answer honestly –No control over participant interpretation Administering –Low response rates –Respondents usually self-selected –Uncertainty about who actually filled out the questionnaire

7. Think about… Reliability –Measures something consistently, but not necessarily what it is supposed to measure –Does NOT imply validity Validity –Measures what it is supposed to measure –Implies reliability (consistency) Measuring your head … intelligence

8. Survey research components Design Sampling Implementation Analysis

9. Survey design Parts of a survey –Introduction –Filtering questions –Content questions –Socio-demographic questions –Debriefing questions Other considerations –Instructions –Formatting

10. The questions Using focus groups –A la Mary Hayden! Borrowing questions –Longitudinal studies Developing them from scratch!

11. Criteria for assessing questions Does the question require an answer? –If you read the area forecast discussion, does it help you better understand the weather? To what extent does the respondent have an accurate, ready-made answer to the question? –What is your favorite episode of Friends? Can people accurately recall and report past behaviors? –What time did you leave work on Dec 20, 2006?

12. More criteria Is the respondent willing to reveal the requested information? –How many times have you driven through flooded roadways? Will the respondent feel motivated to answer each question? –Scenario with 10 subparts

13. Possible scalar categories Strongly agree  strongly disagree Very favorable  very unfavorable Extremely satisfied  extremely dissatisfied None  Extreme Scale of 1 to 5, where 1 means lowest possible quality and 5 means highest possible quality –Scales of 1 to 3, 1 to 7, 1 to 10

14. Question structures Open-ended Close-ended with ordered response categories (like scalar categories) Close-ended with unordered categories Partially close-ended –With “Other (please specify)_________” option

15. Principles for writing questions Choose simple over specialized words Avoid vague quantifiers when more precise estimates can be obtained Avoid specificity that exceeds the respondent’s potential for having an accurate, ready-made answer Use equal numbers of positive and negative categories for scalar questions

16. Principles for writing questions Distinguish undecided from neutral by placement at the end of the scale Avoid bias from unequal comparisons State both sides of attitude scales Avoid check-all-that-apply question formats

17. Principles for writing questions Develop response categories that are mutually exclusive Avoid asking respondents to say yes in order to mean no Avoid double-barreled questions Avoid asking respondents to make unnecessary calculations

18. Pretesting Review by knowledgeable colleagues Evaluate cognitive and motivational qualities –Verbal protocol analysis / think alouds –Retrospective interviews Small pilot study –10% of final sample size Final check

19. How often do you get weather forecasts from the sources listed below? Rarely or never Once or more a month Once a week Two or more times a week Once a day Two or more times a day Local TV stations123456 Cable TV stations (e.g., CNN, The Weather Channel) 123456 Newspapers123456 Telephone (dial-in) weather information source 123456 Commercial or public radio123456 NOAA Weather Radio123456 National Weather Service (NWS) webpages 123456 Other webpages123456 Cell phone, personal desk assistant (PDA), pager, or other electronic device 123456 Friends, family, co-workers, etc. 123456

20. More examples NeverSeldom Some- times UsuallyAlways I watch television or listen to the radio to get a weather forecast so that I can know what to expect 12345

21. More examples I have a plan for keeping myself and my family safe in a tornado. –Yes –No

22. Survey sampling Population, Units, Subjects and Samples –Population: entire group of people about which information wanted. –Units/subjects: Individual members of the population are called units –Sample: part of population examined Error –Sampling: not surveying all elements of population –Coverage: not allowing all members of the survey population to have an equal or nonzero chance of being sampled

23. Survey implementation Methods for survey implementation –Telephone –In-person –Internet Knowledge Networks type of access –Mixed mode (e.g., telephone/mail) –Mail pre-contact cover letter survey instrument return envelope incentive reminder post-card follow-up

24. Survey analysis (examples) Univariate analysis –Histograms, tables, charts, etc. –Mean, median, mode –Range, standard deviation Bivariate analysis –Crosstab/contingency tables –Correlations (Pearson’s r, Spearman’s rho, phi, Cramers V) Regression analysis –Logistic, ordinal, linear, etc. So much more!

25. Final thoughts Many rules Many resources Beg, borrow, and steal NO such thing as a perfect study, survey, or sample!

26. So much more Constraints -- Paperwork Reduction Act, Institutional Review Boards Reliability and validity Errors -- sampling, coverage, measurement, nonresponse Biases

27. Resources Dillman, D.A. 1999. Mail and Internet Surveys: The Tailored Design Method, 2nd Ed. John Wiley Company, New York: NY Krueger, R. A. and M.A. Casey. 2000. Focus Groups: A Practical Guide for Applied Research. SAGE Publications Presser, S., J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer. 2004. Methods for Testing and Evaluating Survey Questionnaires. New York: Wiley and Sons Tourangeau, R., Rips, L.J., and Rasinski, K. 2000. The Psychology of Survey Responses. Cambridge University Press. http://www2.chass.ncsu.edu/garson/pa765/survey.htm http://srcweb.berkeley.edu/index.html http://www.jpsm.umd.edu/jpsm/index.htm EACH OTHER!!