1. Survey Experiments

2. Defined Uses a survey question as its measurement device Manipulates the content, order, format, or other characteristics of the survey as a treatment

3. Methodological Issues Missing Data Matching Both can be an issue in experiments other than surveys

4. Missing Data Some observations missing data on the DV or IVs If missing at random, not a problem to drop from the analysis But usually not missing at random Deleting non-random missing causes bias

5. Missing Data II Data can also be missing intentionally: Some cases not “treated” Possible to “guess” what would have happened to a subject had they been in another treatment group – Allows within-subject comparison of two treatments, the one they received and the one they could have received

6. Solution: Imputation Suppose Y i = a + b 1 X i1 + b 2 X i2 +e i But Y i missing for some observations X i1 and X i2 not missing Regress Y on X i1 and X i2 for all non-missing observations Use b 1 and b 2 to calculate predicted Y p i

7. Better Yet: Multiple Imputation Y p i is a predicted value with uncertainty Multiple imputation predicts multiple values for Y p i drawn from a distribution of predicted values 5 or so predicted Y p i sufficient for inference, no need for many Gary King’s Amelia program available free on- line

8. Matching Experiments can be pre-matched to avoid large random sample Match subjects on important characteristics such as – Sex – Race – Age – Education levels – Other traits?

10. Matching Often necessary in field experiment when randomization more difficult to control propensity score is the probability of an observation being assigned to a particular treatment in a study given a set of known variables. Propensity scores reduce selection bias by equating groups based on these variables

11. A Theory of Nonseparable Preferences in Survey Responses

12. Question Why do people change their answers to survey questions if the order of questions changes? Does changing survey responses indicate that people do not have well-formed opinions

14. Theory Nonseparable Preferences: What a person wants on one issue depends on what she gets on another issue Separable Preferences: What a person wants on every issue is independent of what they get on other issues

15. Measuring Nonseparable Preferences

19. Method Randomize the order of pairs of survey questions – For some issues, aggregate responses different across question order Each subject answers questions in order – Issue 1 then Issue 2 – Issue 2 then Issue 1

20. Method Impute what subject would have answered had they heard questions in different order For each question we then have Y i (if first) – Y i (if second) One of these will be imputed for each person since they cannot answer a question both first and second in the order First study to analyze individual differences in question orders, not simply aggregate differences

22. Conclusions Nonseparable preferences explain question order effects Political information level does not Response instability not due to uninformed respondents

23. Are Survey Experiments Externally Valid? JASON BARABAS and JENNIFER JERIT American Political Science Review 2010

24. Question Many survey experiments expose subjects to different information to show effect of on responses In a survey experiment, subjects are a “captive audience” that must pay attention Do the same information effects appear in the real world Compare survey experiments with natural experiments

25. Method Survey experiments give people to political information about immigration and medical care Pre-post survey also in field during change in medical insurance and immigration – Ask respondents which media sources they use Is the effect of information in the survey experiment as large as in the natural experiment?