1. Survey Research: TTM531 Class Presentation 1 SURVEY RESEARCH TTM531 Fall 2001 Class Presentation Nancy Whelchel Coordinator for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184 http://www2.acs.ncsu.edu/UPA/survey/survey_research/

2. Survey Research: TTM531 Class Presentation 2 Major Tasks l ID Purpose of Research l Decide on Survey Design and Format l Select Sample l Design Survey Instrument l Pre-test Instrument l Collect Data l Analyze Data l Report Findings

3. Survey Research: TTM531 Class Presentation 3 Purpose of Your Survey Ask Yourself: l WHY do I want to/need to do a survey? l WHAT do I want to learn? l WHO is my audience? l HOW will I use the information I gather?

4. Survey Research: TTM531 Class Presentation 4 Purpose of Your Survey Our Example: l Why = President of NCTextiles.com, where you work, told you to. l What want to learn = How does employee satisfaction and productivity compare between NCTextiles.com telecommuters and non- telecommuters. l Who is audience = Company president and executive board. l How use = Set new policies for telecommuting.

5. Survey Research: TTM531 Class Presentation 5 Purpose of Your Survey l Description Discover the distribution of certain traits or characteristics (e.g., #telecommuters, $revenue generated) l Explanation Understand why something happens (e.g., is there a difference between telecommuters and non-telecommuters in job satisfaction and in productivity) l Exploration Help inform a research design l

6. Survey Research: TTM531 Class Presentation 6 Think About Available Resources l What information is already available from non-survey sources? l Have any surveys already been done on this question? l Who is going to: Design the survey Administer the survey Enter the data Analyze the data Write up the results Present the findings

7. Survey Research: TTM531 Class Presentation 7 Think About Available Resources Should I hire a professional??? l What resources/skills are available in- house? l What types of decisions are going to be made based on your findings? l How much money do you have for the survey?

8. Survey Research: TTM531 Class Presentation 8 Basic Survey Designs l Cross-sectional Data collected at one point in time from a sample selected to describe a larger population l Longitudinal Trend Different samples from a general population studied at different points in time. Cohort Same specific population studied over time (may be different samples). Panel Same sample studied over time.

9. Survey Research: TTM531 Class Presentation 9 Basic Survey Formats Format determines much of research design (e.g., sampling, questionnaire design, administration, data analysis, etc.) Basic formats include: Self-administered mail survey Telephone interview Face-to-face interview Focus group Self-administered Web survey Self-administered Email survey

10. Survey Research: TTM531 Class Presentation 10 The Survey Sample WHO or WHAT do I want to survey? Sampling allows you to study just a portion of the population/group and make estimated assertions about the population from the sample. Warning: Bad Sample = Bad Data

11. Survey Research: TTM531 Class Presentation 11 Why Sample? l Time Shorter time in the field decreases the chance that external factors will influence results. l Quality Doing fewer surveys increases control over the entire process (interviewing, data entry, etc.) and therefore decreases error l Cost

12. Survey Research: TTM531 Class Presentation 12 What is a Sample? Some definitions: l Element or Sampling Unit Unit about which information is collected (e.g., employee) l Survey Population Aggregate of all elements interested in from which sample is selected (must be clearly defined) (e.g., NCTextiles.com non-executive employees with job tenure 3+ months )

13. Survey Research: TTM531 Class Presentation 13 What is a Sample? Some more definitions: l Sampling Frame Actual physical source listing elements in the survey population e.g., Edited list from company employee directory. l Sample Population Collection of elements selected from the survey population sampling frame

14. Survey Research: TTM531 Class Presentation 14 What is a Sample? A Warning: Findings from a survey are only representative of the aggregate of elements in the sampling frame. In other words… You can’t use your results to say anything about the folks who didn’t have a chance to be included.

15. Survey Research: TTM531 Class Presentation 15 Sample Size Depends on: l Cost l Level of precision want (“margin of error”) l Level of confidence want l Anticipated response rate l Number of responses needed per subgroup interested in

16. Survey Research: TTM531 Class Presentation 16 Sample Size Example: Margin of Error +/- points confidence interval 80% 90% 95% 5 150 275 400 10 40 70 100 If 45% of 400 respondents agree that NCTextiles.com provides “excellent” health benefits, this means that 95 out of 100 times between 40% (45%-5) and 50% (45%+5) believe the company provides “excellent” health benefits. Source: McClelland, Lou (1993), “Survey Research Mechanics in Three Checklists,” (working paper). Office of Research and Information: University of Colorado at Boulder.

17. Survey Research: TTM531 Class Presentation 17 Selecting a Sample l Probability Sample A representative sample Everyone in the sample population has an equal chance of getting in the sample. Results can be generalized to the sample population. l Non-Probability Sample Non-representative sample Purposive, quota, who’s available Results can not be generalized beyond the sample.

18. Survey Research: TTM531 Class Presentation 18 Probability Sampling l Simple random sample List the elements in the sampling frame and use a random number chart to select elements. l Systematic sampling List the elements in the sampling frame and take every Nth element, based on how many needed in sample. Warning: Be attentive to any inherent patterns in how sampling frame is ordered.

19. Survey Research: TTM531 Class Presentation 19 Probability Sampling l Stratified sampling Draw appropriate proportion of elements from homogeneous groups interested in (based on theory). Note: Think about using disproportionate sampling then “weighting” the data. E.G., If there are relatively few numbers in one sub-group you are interested in, include proportionally more of them in your sample, then “weight” the data to reflect their true proportion in your population.

20. Survey Research: TTM531 Class Presentation 20 Probability Sampling l Stratified sampling continued An Example: 250 out of your 2000 employees in the sample population telecommute (12.5% of sample population). With stratified sampling with sample size of 800, would have 100 telecommuters (800 *.125), and 700 non-telecommuters (800 *.875).

21. Survey Research: TTM531 Class Presentation 21 Probability Sampling l Stratified sampling continued An Example continued: BUT, you might want to “oversample” telecommuters (even include all 250) to be able have less error in their results, and “undersample” non-telecommuters (e.g., 550). When analyzing all results, “weight down” telecommuters to reflect their true proportion in the sample population, and “weight up” non- telecommuters.

22. Survey Research: TTM531 Class Presentation 22 Instrument Design Start with: l Conceptualization Define the concepts you want to study l Operationalization Specify indicators of your concepts. Make exhaustive list of everything the concept could include

23. Survey Research: TTM531 Class Presentation 23 Instrument Design Conceptualization Job Satisfaction = “A pleasurable or positive emotional state resulting from the appraisal of one’s job.” Includes both extrinsic and intrinsic aspects. Has several dimensions: importance to employee motivation of employee current level of attainment satisfaction with attainment

24. Survey Research: TTM531 Class Presentation 24 Instrument Design Extrinsic: pay promotion supervision co-workers company physical conditions security social status work itself Intrinsic: variety of work autonomy and freedom responsibility feedback creativity recognition meaningfulness of work Operationalization: Job Statisfaction

25. Survey Research: TTM531 Class Presentation 25 Questionnaire Construction The way a question or statement is worded and the response options offered determine the nature of the data received. Bad question = Bad data

26. Survey Research: TTM531 Class Presentation 26 Questionnaire Construction Some jargon… l Questions need to be VALID “…an empirical measure adequately reflects the real meaning of the concept under consideration.” (Babbie, 1990) face validity = the measure jibes with common agreement of what a concept entails. predictive validity = the measure accurately predicts behavior content validity = the measure covers the range of meanings included within the concept. construct validity = the measure relates to other measures theoretically related to concept.

27. Survey Research: TTM531 Class Presentation 27 Questionnaire Construction Some more jargon… l Questions need to be RELIABLE “…a particular technique, applied repeatedly to the same object, would yield the same result each time.” (Babbie, 1990)

28. Survey Research: TTM531 Class Presentation 28 Questionnaire Construction Open-end Questions Respondents write question response in own words probably need to enter data by hand develop coding scheme for responses content analysis ? Suggestion: It’s a good idea to always include an open-end question giving the respondent the opportunity to add any additional comments they might have.

29. Survey Research: TTM531 Class Presentation 29 Questionnaire Construction Closed-end or Forced-choice Questions Respondent selects question answer from among a list of possible responses Suggestion: It’s a good idea to ask multiple indicators of a single concept, and then create a scale by combining those indicators.

30. Survey Research: TTM531 Class Presentation 30 Good Questions... l Are clear and use simple language l Are concise l Are specific l Are possible to answer l Are relevant to the respondent l Do not use negatives l Avoid biased terms l Have only 1 part (not “double-barrel”)

31. Survey Research: TTM531 Class Presentation 31 Good Questions... Use simple language... My job is usually interesting enough to keep me from getting bored. Is better than : My job is usually engaging enough to permit me from reaching a banausic state of mind.

32. Survey Research: TTM531 Class Presentation 32 Good Questions... Are concise... How interested would you be in a flexible work schedule in which you would have to work an 8 hour day in the office, but would be able to arrive at work anytime between 6:00 and 10:00 AM and leave between 2:00 and 7:00?. Is better than: How interested would you be in being able to leave your house either before the morning rush hour or after the morning rush hour, and then arrive at the office not at the standard 8:00 AM time that is required according to the employee handbook, but at whatever time you got there, and then work a 8 hour day at the office (excluding and hour for lunch), and then leave the office so you are not commuting during rush hour but either before or after rush hour?

33. Survey Research: TTM531 Class Presentation 33 Good Questions... Are specific... In what year did you first start working at NCTextiles.com? Is better than: When did you start working here?

34. Survey Research: TTM531 Class Presentation 34 Good Questions... Are possible for the respondent to answer... NO: My contribution to the company has had a direct impact on its ability to control a broader share in the global market.

35. Survey Research: TTM531 Class Presentation 35 Good Questions... Are relevant... NO: Polyester doesn’t breathe as well as natural fibers.

36. Survey Research: TTM531 Class Presentation 36 Good Questions... Do not use negatives... The lack of flexibility in my work hours makes it difficult for me to spend the amount of time I would like to with my family. Is better than: The lack of flexibility in my work hours does not make it possible for me to spend the amount of time I would like to with my family.

37. Survey Research: TTM531 Class Presentation 37 Good Questions... Do not use biased terms or suggestions... Do you agree or disagree with the following statement: All employees need to be in the office during regular business hours. Is better than: Don’t you agree with the president of the company when she says that all employees need to be in the office during regular business hours?

38. Survey Research: TTM531 Class Presentation 38 Good Questions.. Do not ask two questions in one (“double- barrel”)... I find real enjoyment in the work I do. I like the hours I keep at work. Is better than: I find real enjoyment in the work I do and the hours I keep.

39. Survey Research: TTM531 Class Presentation 39 Response Options Response options should reflect the concepts you are trying to measure, and fit with the question wording. Suggestion: Avoid simple “yes” or “no” type responses and try to measure intensity of feeling when possible.

40. Survey Research: TTM531 Class Presentation 40 Response Options Must be: l Mutually exclusive can select only one appropriate answer Example: NOT mutually exclusive... Q. On most days, what time do you leave your house for work? 6:00 AM or earlier 7:30-8:00 AM 6:00-6:30 AM 8:00-8:30 AM 6:30-7:00 AM 8:30-9:00 AM 7:00-7:30 AM 9:00 AM or later

41. Survey Research: TTM531 Class Presentation 41 Response Options l Mutually exclusive continued Example: IS mutually exclusive... Q. On most days, what time do you leave your house for work? Before 6:00 AM 7:30-7:59 AM 6:00-6:29 AM 8:00-8:29 AM 6:30-6:59 AM 8:30-8:59 AM 7:00-7:29 AM 9:00 AM or later

42. Survey Research: TTM531 Class Presentation 42 Response Options Must be: l Exhaustive all possible answers are listed (including e.g., “other, “ “don’t know,” etc.) Example: NOT exhaustive Q. On most days, what time do you leave your house for work? 6:00-6:29 AM 7:30-7:59 AM 6:30-6:59 AM 8:00-8:29 AM 7:00-7:29 AM 8:30-8:59 AM

43. Survey Research: TTM531 Class Presentation 43 Response Options l Exhaustive continued Example: IS exhaustive Q. On most days, what time do you leave your house for work? Before 6:00 AM 7:30-7:59 AM 6:00-6:29 AM 8:00-8:29 AM 6:30-6:59 AM 8:30-8:59 AM 7:00-7:29 AM 9:00 AM or later

44. Survey Research: TTM531 Class Presentation 44 Question Order l Be attentive: Initial questions effect answers to subsequent ones. l Ask less interesting questions (e.g., demographics) at end of self- administered survey and beginning of interview survey.

45. Survey Research: TTM531 Class Presentation 45 Instructions for Respondent l Provide general instructions in beginning of self-administered survey Brief explanation could include: why doing survey why answers are important how to answer questions stress confidentiality (if appropriate) Do NOT ask permission - - assume respondent will want to participate. l Provide specific instructions to questions as necessary

46. Survey Research: TTM531 Class Presentation 46 Instructions for Respondent Example Instructions: The following statements are about some features of your job. Please tell us how satisfied or dissatisfied you feel with the features of your present job by circling one of the response options listed below each statement. There are no right or wrong answers. All replies are confidential so give your honest opinion on each of the statements. Please do not discuss your answers with other people. It is your opinion which matters. Even though it may be hard to decide, please be sure to answer all questions. Thank you for your help.

47. Survey Research: TTM531 Class Presentation 47 Questionnaire Format l Should be visually attractive and nicely reproduced l Should be spread out and uncluttered better to have more than 1 page than cramped page l Should be broken into logical sections when possible l Should have clear skip patterns for contingency questions

48. Survey Research: TTM531 Class Presentation 48 Questionnaire Format l Should have clear spaces for respondents to mark answers use boxes parentheses ( X ) or numbers to circle avoid lines to put check on X_ X 1

49. Survey Research: TTM531 Class Presentation 49 Questionnaire Format Helpful hint…. Be sure to seek advice on question wording, response options, and questionnaire format from those who will be doing the data entry and data analyses.

50. Survey Research: TTM531 Class Presentation 50 Pre-Test Test the survey instrument before doing the actual study. Suggestion: It’s also a good idea to test out your sample design, data collection methods, data processing, and data analysis if possible.

51. Survey Research: TTM531 Class Presentation 51 Pre-Testing the Survey Instrument l Use respondents similar to those who will be in survey sample. l Test: Question clarity failure to answer? multiple answers? “other” answers? qualified answers?

52. Survey Research: TTM531 Class Presentation 52 Pre-Testing the Survey Instrument l Test continued : Questionnaire format instructions? flow? layout? Variance in responses (if interested in relationships among variables…) do most respondents give the same answer to a question?

53. Survey Research: TTM531 Class Presentation 53 Pre-Testing the Survey Instrument l Revise survey instrument as necessary l Pre-test again if possible (ideally with some of original pre-test respondents, and some new respondents)

54. Survey Research: TTM531 Class Presentation 54 Data Collection (Self-administered mail survey) Include: l Survey instrument l Return envelope self-addressed and stamped (either postage-paid or 1st class stamp) l Cover letter on letterhead paper signed by most important person possible

55. Survey Research: TTM531 Class Presentation 55 Data Collection (Self-administered mail survey) l Cover letter (continued) brief explanation of purpose of survey how results will be used why respondent was selected why all answers are important emphasize confidentiality (if applicable) provide contact information for respondents with questions about the survey Describe/explain any incentives for participating in the survey and/or consequences for non-participation if applicable.

56. Survey Research: TTM531 Class Presentation 56 Identifying Respondents and Non-Respondents If respondents are not asked to provide any identifying information on the survey, you can… Number the surveys before they are sent out OR Unobtrusively number the return envelopes before they are sent in the survey packet AND Keep a list of survey/envelope numbers corresponding to respondent name or id.

57. Survey Research: TTM531 Class Presentation 57 Monitor Returned Surveys l Open surveys as they are returned l Assign unique ID number to each survey (if not already assigned) l Stamp date on survey l Track the number of survey coming in each day Use this information to help inform timing of follow-up reminders.

58. Survey Research: TTM531 Class Presentation 58 Follow-up Reminders Sending 2 follow-up reminders works best to increase response rates. l Timing first follow-up 2-3 weeks after original second follow-up 2-3 weeks after 1st follow-up l Materials reminder card with information to have new survey sent if necessary OR entire survey packet

59. Survey Research: TTM531 Class Presentation 59 Follow-up Reminders l Send to: Non-respondents only if possible to identify All respondents if not possible to identify non- respondents include “if you have already responded, thank you for participating….”

60. Survey Research: TTM531 Class Presentation 60 Response Rates Response Rate = n surveys completed and returned net sample size (net sample size = sample size - undeliverable surveys) In general … Acceptable response rate = 50% Good response rate = 60% Very good response rate = 70%

61. Survey Research: TTM531 Class Presentation 61 And then…. l Data Entry l Data Analysis l Report Writing l Presentation

62. Survey Research: TTM531 Class Presentation 62 Some References Babbie, Earl (1990), Survey Research Methods. Wadsworth Publishing: CA. Cox, B., et. al., (1995), Business Survey Methods. Wiley: NY. Dillman, D.A. (1991), “The Design and Administration of Mail Surveys.” Annual Review of Sociology, 17:225-249. Fink, Arlene (1995), How to Sample in Surveys. Sage: CA. Fowler, Floyd J. (1993), Survey Research Methods. Sage: CA. Fowler, Floyd J. (1995), Improving Survey Questions. Sage: CA. Sudman, Seymour and Bradburn, Norman (1982), A Practical Guide to Questionnaire Design. Jossey-Bass: San Francisco. Tourangeau, R., and Smith, T.W. (1996), “Asking Senstive Questions: The Impact of Data Collection Mode, Question Format, and Question Context, “ Public Opinion Quarterly, 60:275-304. See also: Course Descriptions and Syllabi: the Joint Program in Survey Methodology (University of Maryland, University of Michigan, and Westat) at http://www.jpsm.umd.edu/classinfo.htm