1. Survey Research Methods Jeff Lazo (please don’t forget I am an economist after all) (with input from Julie, Mary Hayden, and many many others... ) WAS*IS Boulder, CO August 11, 2008

2. Asking the question Did you kill your wife? (check all that apply)  Yes  No  Maybe  Don’t Know  None of the above  All of the above  Was that MY wife? Did you happen to have murdered your wife? As you know, many people kill their wives nowadays. Did you happen to have killed yours? Do you know about other people who have killed their wives? How about yourself? Thank you for completing this survey, and by the way, did you kill your wife? Three cards are attached to this survey. One says your wife died of natural causes; one says you killed her; and the third says Other (explain). Please tear off the cards that do not apply, leaving the one that best describes your situation. http://faculty.ncwc.edu/toconnor/308/308lect07.htm

3. Weather Comprehension Survey Q1 Do you know what a warning means?  Yes  No Q1a Do you know what a warning means?  Yes – and don’t expect me to explain my answer  Yes - because I flipped my yes/no coin and it came up “Yes”  Yes – because if I answer “yes” you’ll leave me alone.  Yes – because based on that look on your face, I suspect that is the right answer.  Yes – because if I didn’t I would probably look stupid.  Yes – because that is the first response option  Yes – a warning means “ A warning is issued when a hazardous weather or hydrologic event is occurring, is imminent, or has a very high probability of occurring. A warning is used for conditions posing a threat to life or property ” 1  No - because I flipped my yes/no coin and it came up “No”  No – because it depends on what you mean by “mean”  No – because I really don’t know what a warning means...  No – and I still say I didn’t kill my wife... 1 http://www.crh.noaa.gov/glossary.php?letter=w

4. Observations and Data Assimilation... and modeling and forecasting

5. Survey experience Studies Arkansas River Inter-basin Water Transfers groundwater, spotted owls, coal dust, global warming & recreational fishing, global warming & duck hunting, Green Bay, Kalamazoo, Hudson, air toxics, Confidence, Contra Costa, Hospital, Reliable, Storm, Hurricane, Communicating Forecast Uncertainty, Weather and Society Watch Readers, Damage Data, WAS*IS Assessment Development focus-groups cognitive interviews Implementation mail telephone internet focus-group site written mixed methods And I really really really still don’t know any “best” way to do survey work...

6. To Survey or Not to Survey – That is the question... Problem definition... –What question am I trying to answer? Objectives –What information do I need? –Who do I need it from? –How much information do I need? –What will I do with the information? –How will I analyze the information? –How and to whom will I present the results?

7. Different types of surveys attitudinal behavioral information gathering (economic surveys) valuation

8. Advantages to surveys Gathering Information –Efficient for collecting large amount of information –Flexible to collect wide range of information attitudes, values, beliefs, and past behaviors. Questions –Standardized - relatively free from several types of errors –Researcher can ask more complex questions –Diminished response effect willing to divulge more info w/o face-to-face contact less likely to try to impress interviewer) Administration –Relatively easy to administer Computer-based – automatic data entry –Economy in data collection Analysis –Statistical techniques to determine validity, reliability, and statistical significance http://en.wikipedia.org/wiki/Statistical_survey

9. Disadvantages to surveys Gathering Information –Questions must be general enough for all respondents –Non-literate or illiterate populations only English-speaking populations hard-to-reach populations – “vulnerable populations” Questions –Can’t determine subjects’ motivation, memory, and ability to respond –Not appropriate for studying complex social phenomena –Subjects may not answer honestly –No control over participant interpretation Administration –Low response rates –Respondents usually self-selected Uncertainty about who actually filled out the questionnaire Analysis –Statistical techniques to determine validity, reliability, and statistical significance

10. Constraints on survey research Paperwork Reduction Act other regulatory constraints human subjects - Institutional Review Board costs and time resources for survey research –natural resources damage assessment (NRDA) –grad students and pizzas

11. Survey biases social desirability bias interviewer bias non-response bias item non-response measurement bias hypothetical bias information bias

12. Reliability and Validity Population parameter (Red line) = Sample statistic (Yellow line) --> unbiased High variance (Green line) Unreliable but valid http://seamonkey.ed.asu.edu/~alex/teaching/assessment/reliability.html

13. Reliability and Validity Population parameter (Red line) <> Sample statistic (Yellow line) Biased - low variance (Green line) Invalid but reliable

14. Figure 1: Survey Design and Implementation Procedures Fritz Scheuren. “What is a Survey.” http://www.amstat.org/sections/srms/pamphlet.pdf accessed 10.01.04).http://www.amstat.org/sections/srms/pamphlet.pdf

15. Survey research – components Survey design Sampling Implementation Analysis and reporting

16. Survey design Technical background Preliminary survey design introduction –tell respondents purpose of the survey providing information how to ask the correct questions response categories socio-demographic information de-briefing questions

17. Survey design Focus-groups Also important as qualitative research tool Ex: focus-group 9/10/2001 – Poughkeepsie, NY One-on-one cognitive interviews verbal protocols –think alouds retrospective reports Pre-testing survey survey instrument implementation method analysis planning

18. Survey design Parts of a survey –Introduction –Filtering questions –Content questions –Socio-demographic questions –Debriefing questions Other considerations –Instructions –Formatting

19. Peer review Research objective... technical information sampling plan Implementation plan formatting and presentation questions and survey language –Uncle George test analysis plan

20. Sampling Population, Units, Subjects and Samples –Population: entire group of people about which information wanted. –Units/subjects: Individual members of the population are called units –Sample: part of population examined Error –Sampling: not surveying all elements of population –Coverage: not allowing all members of the survey population to have an equal or nonzero chance of being sampled http://www.gseis.ucla.edu/courses/ed230a2/notes2/sampling.html

21. Sampling Types of Samples –Voluntary response sample –Convenience (accidental, haphazard) sample –Simple random sample –Stratified random sample –Cluster random sample –Multistage sample design

22. Implementation methods for survey implementation –telephone –in-person –internet Knowledge Networks type of access ResearchExec –mixed mode – e.g., telephone/mail –mail the mail survey “package” – Dillman Total Design Method (TDM) –pre-contact –cover letter –survey instrument –return envelope –incentive –reminder post-card –follow-up

23. Analysis and reporting QA/QC response rate – AAPOR socio-demographics of pop / sample / respondents analysis methods –content analysis –factor analysis –latent class variable analysis quantitative analysis –summary data –basic statistical analysis –econometric modeling

24. Survey analysis (examples) Univariate analysis –Histograms, tables, charts, etc. –Mean, median, mode –Range, standard deviation Bivariate analysis –Crosstab/contingency tables –Correlations (Pearson’s r, Spearman’s rho, phi, Cramers V) Regression analysis –Logistic, ordinal, linear, etc. So much more!

25. Conclusions... But still want to look at an example to follow... Many rules Many resources Beg, borrow, and steal! No such thing as perfect sample or survey or study - so, sometimes better to plow ahead

26. Literature Dillman, D.A. 1999. Mail and Internet Surveys: The Tailored Design Method, 2nd Ed. John Wiley Company, New York: NY Ericsson, K.A. and H.A. Simon. 1993. Protocol Analysis: Verbal Reports as Data. Bradford Book, MIT Press, Cambridge, MA. Krueger, R. A. and M.A. Casey. 2000. Focus Groups: A Practical Guide for Applied Research. SAGE Publications Presser, S., J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer. 2004. Methods for Testing and Evaluating Survey Questionnaires. New York: Wiley Tourangeau, R., Rips, L.J., and Rasinski, K. 2000. The Psychology of Survey Responses. Cambridge University Press.

27. Resources The American Association for Public Opinion Research: http://www.aapor.org/ http://www.aapor.org/ Survey Research Center –University of California, Berkeley: http://srcweb.berkeley.edu/index.html http://srcweb.berkeley.edu/index.html Survey Research Center –Centers of the Institute for Social Research (ISR). – University of Michigan: http://www.isr.umich.edu/src/http://www.isr.umich.edu/src/ Joint Program in Survey Methodology (JPSM) –University of Maryland. http://www.jpsm.umd.edu/jpsm/index.htm http://www.jpsm.umd.edu/jpsm/index.htm

28. Asking the question

29. More useful info don’t have time for this time around...

30. Principles for writing questions Choose simple over specialized words Avoid vague quantifiers when more precise estimates can be obtained Avoid specificity that exceeds the respondent’s potential for having an accurate, ready-made answer Use equal numbers of positive and negative categories for scalar questions

31. Principles for writing questions Distinguish undecided from neutral by placement at the end of the scale Avoid bias from unequal comparisons State both sides of attitude scales Avoid check-all-that-apply question formats

32. Principles for writing questions Develop response categories that are mutually exclusive Avoid asking respondents to say yes in order to mean no Avoid double-barreled questions Avoid asking respondents to make unnecessary calculations

33. Pretesting Review by knowledgeable colleagues Evaluate cognitive and motivational qualities –Verbal protocol analysis / think alouds –Retrospective interviews Small pilot study –10% of final sample size Final check