Lecture 4 Survey Design and Data Coding

2 Overview What is a survey Question design and considerations Testing a survey instrument Data considerations (data coding)

3 What is a Survey? To survey: “act of looking or seeing, observing” “act of looking or seeing, observing” Research Surveys Qualitative interviews, focus groups Qualitative interviews, focus groups Specific, systematic, quantitative data- collection instruments Specific, systematic, quantitative data- collection instruments

4

5 Example Surveys US Census US Census General Social Survey General Social Survey Online Surveys Online Surveys

6 Some General Considerations for a Survey Must have a logic for each question Must have a logic for the question responses (if provided) Must have a logic for the question responses (if provided) Must have a logic for the sequencing of questions Must have a logic for the sequencing of questions Must have clear wording Must have clear formatting Must take into account the sample population that will actually take/use the survey instrument Culture, language, interpretive ambiguity Culture, language, interpretive ambiguity

7 Question Logic Avoid redundant questions unless you have a reason Exception: Sometimes it is a good idea to ask two questions that tap same concept (first as a scale, then as a categorical decision) Exception: Sometimes it is a good idea to ask two questions that tap same concept (first as a scale, then as a categorical decision) Example (ranks): First ask respondent to rate 1-10 how much they agree with several statements. First ask respondent to rate 1-10 how much they agree with several statements. Next, ask respondents to rank the statements by how much they agree with them (i.e., 5 statements, rank them 1-5). Next, ask respondents to rank the statements by how much they agree with them (i.e., 5 statements, rank them 1-5).

8 Question Logic (continued) Avoid asking unnecessary questions Example: Survey on computer usage Example: Survey on computer usage Socio-demographic questions (age, gender)? Risk behavior questions (drug use, etc)?

9 Question Response Logic: Scales versus Categories As a basic rule, metric scales with more range are better than binary or categorical responses (when appropriate). Example: Happiness Example: Happiness Are you happy or sad today? How happy are you on a scale of 0-10 (0 is least happy, 10 is most happy)? With general scales and Likert-scales, consider having no “middle” category (neutral, no opinion).

10 Question Response Logic: Categorical Responses Responses must be mutually exclusive Example (bad): Where do you live? Example (bad): Where do you live? Berkeley, San Fran, Bay Area, Other Berkeley, San Fran, Bay Area, Other Responses must be exhaustive Example (bad): What kind of computer do you have? Example (bad): What kind of computer do you have? PC, Mac Use ‘Don’t Know’, ‘Other’, ‘Not Applicable’ when absolutely necessary

11 Question Sequence Static order for questions needs to have some rationale/logic Grouping similar items together Grouping similar items together Scattering similar items throughout survey Scattering similar items throughout survey Personal demographic questions work best at end of survey (response rate and completion) Randomization for all respondents

12 Clear Wording / Leading Questions Clear Wording Example: Example: “What ISP do you use?” “What ISP do you use?” “If you have Internet service at home, what company or service provider do you use for Internet access?” Leading Questions Example: Example: “Don’t you think we should support our troops in Iraq?” “How strongly do you agree or disagree with the following question: ‘We should support our troops in Iraq’”

13 Clear Formatting, Logic Not all questions apply to everyone Example: Example: “How much do you spend on gas heat each month?” Branching is a possible solution Example: Example: “Do you have gas heat? If yes, go to next question. If not, skip to question #3. Condense when possible to avoid unnecessary branching. Example: Example: “How much do you spend on gas heat each month? (write 0 if you do not have gas heat)

14 Know your sample population Regional language and terminology Cultural differences How you conduct survey can influence your valid sample Door to door? Door to door? Registered telephone directory? Registered telephone directory? Internet-based survey? Internet-based survey?

15 Replication and Using Existing Survey Instruments ALWAYS a good idea to find other surveys that are used in your area of interest. Especially with large, funded surveys the questions may have been tested for reliability. Especially with large, funded surveys the questions may have been tested for reliability. Allows for comparisons between different samples if the question wording is the same. Allows for comparisons between different samples if the question wording is the same. If a question or set of questions is accepted as a good operationalization of the concept you are interested in, you don’t want to reinvent it unless you really intend to argue that your measure is more appropriate. If a question or set of questions is accepted as a good operationalization of the concept you are interested in, you don’t want to reinvent it unless you really intend to argue that your measure is more appropriate.

Pre-Testing and Pilot Studies

17 Testing a Survey Instrument Pre-testing versus Pilots Pre-tests: Focus on individual questions or the entire survey instrument/questionnaire. Pre-tests: Focus on individual questions or the entire survey instrument/questionnaire. Pilots: Usually larger scale than pre-testing, involve testing the entire survey procedure. Pilots: Usually larger scale than pre-testing, involve testing the entire survey procedure.

18 Pre-Testing and Pilots Pre-tests and Pilots are always necessary, unless the survey in its existing form has already been given before. Pre-testing and Pilot studies should have a large enough response rate so that you can actually find problems! Example: You want to survey 100 undergrads for a small study. You may need to at least pre-test on a 20% sample with different undergrads than your intended valid sample (i.e., pre- test on 20 undergrads from your intended population, but not students who could end up in your final survey) Example: You want to survey 100 undergrads for a small study. You may need to at least pre-test on a 20% sample with different undergrads than your intended valid sample (i.e., pre- test on 20 undergrads from your intended population, but not students who could end up in your final survey)

19 Testing your Questions Does the respondent’s comprehension of question meaning match that of the researcher? Does the researcher put too much of an expectation of recall on the respondent?

20 Ways to test your questions Behavior coding– interview some respondents as you give the survey questions and keep track of requests for clarification. Ask pretest respondents to rephrase your questions in their own words. Panels of ‘experts’: give your questions to groups of individuals for comments/suggestions.

21 The pitfalls of skipping the testing stage Best case scenario: you get some or most of what you wanted to get– but often an uphill battle with justifying your operationalizations and wording choices. Worst case scenario: you get wild differences in responses, respondents don’t understand key questions, large incompletion rate, money and time spent on conducting survey is wasted (except for your newfound appreciation for pre- testing and pilots)

After the Survey: Data Coding and Error Checking

23 Data Coding The coding considerations start with the survey itself. You develop a codebook that records what the possible numeric responses will be for every question. No open-ended questions unless absolutely necessary for other reasons.

24 Making data numeric Use numbers to represent variable values Assign a numeric value to all of the values that your variables can take. Assign a numeric value to all of the values that your variables can take. Example: Gender (Male, Female) Male=0, Female=1. Develop a systematic way of handling missing data! You must enter a value for missing data– otherwise you will not know if missing is due to input error, N/A, skipped question, etc. You must enter a value for missing data– otherwise you will not know if missing is due to input error, N/A, skipped question, etc. Example: use numeric codes that would not normally make sense for the variable (e.g., -9 for Missing, -8 for Not Applicable, etc).

25 Other tips for creating your dataset Use ID numbers– always, no exceptions! seriously! Datasets get manipulated and resorted constantly. Without ID’s, errors cannot be corrected, outliers cannot be identified. ID’s should allow you to match any case in the dataset with an actual survey taken by that individual. Datasets get manipulated and resorted constantly. Without ID’s, errors cannot be corrected, outliers cannot be identified. ID’s should allow you to match any case in the dataset with an actual survey taken by that individual. Use conventional data structure. Rectangular format, each row is a case and each column is a single variable. Rectangular format, each row is a case and each column is a single variable.

26 Error Checking Data Why? Solves problems that may occur later Solves problems that may occur later Makes sure your entire analysis is not bogus Makes sure your entire analysis is not bogus You may accidentally engage in coitus more often as you get older….WHAT?! You may accidentally engage in coitus more often as you get older….WHAT?!

27 Marital Coital Frequency Jasso and Guillermina (1985) “Marital Coital Frequency and the Passage of Time: Estimating the Separate Effects of Spouses’ Ages and Marital Duration, Birth and Marriage Cohorts, and Period Influences” (American Sociological Review) Major Findings of the Study: Major Findings of the Study: Controlling for cohort and age effects, negative period effect Controlling for period and cohort effects, wife’s age had a positive effect Both findings differ significantly from earlier studies of the same topic.

28 Coitus: Part Deux Kahn and Udry (1986) “Marital Coital Frequency: Unnoticed Outliers and Unspecified Interactions Lead to Erroneous Conclusions” (American Sociological Review) Major Findings: Major Findings: In the Jasso study, 4 cases were coded as 88– MISSING DATA CODES!!! 4 more cases had very large studentized residuals (each was also very different from the first survey) Missed an important interaction between length of marriage and wife’s age Dropping the 8 outliers from the sample of more than 2000 cases drastically changed the findings

29 How to Error Check data Know what you are looking to check, use appropriate methods: Descriptives Descriptives Frequencies Frequencies Cross-tabulations Cross-tabulations

30 Error Checking Examples Checking Original Variables for Errors Frequencies Frequencies Descriptives Descriptives Checking and Setting “Missing” codes Recoding and Creating New Variables from Existing Variables Frequencies Frequencies Cross-Tabulations Cross-Tabulations

31 Example: Class Data Set