Survey Experiments

Defined Uses a survey question as its measurement device Manipulates the content, order, format, or other characteristics of the survey as a treatment

Methodological Issues Missing Data Matching Both can be an issue in experiments other than surveys

Missing Data Some observations missing data on the DV or IVs If missing at random, not a problem to drop from the analysis But usually not missing at random Deleting non-random missing causes bias

Missing Data II Data can also be missing intentionally: Some cases not “treated” Possible to “guess” what would have happened to a subject had they been in another treatment group – Allows within-subject comparison of two treatments, the one they received and the one they could have received

Solution: Imputation Suppose Y i = a + b 1 X i1 + b 2 X i2 +e i But Y i missing for some observations X i1 and X i2 not missing Regress Y on X i1 and X i2 for all non-missing observations Use b 1 and b 2 to calculate predicted Y p i

Better Yet: Multiple Imputation Y p i is a predicted value with uncertainty Multiple imputation predicts multiple values for Y p i drawn from a distribution of predicted values 5 or so predicted Y p i sufficient for inference, no need for many Gary King’s Amelia program available free on- line

Matching Experiments can be pre-matched to avoid large random sample Match subjects on important characteristics such as – Sex – Race – Age – Education levels – Other traits?

Matching Often necessary in field experiment when randomization more difficult to control propensity score is the probability of an observation being assigned to a particular treatment in a study given a set of known variables. Propensity scores reduce selection bias by equating groups based on these variables

A Theory of Nonseparable Preferences in Survey Responses

Question Why do people change their answers to survey questions if the order of questions changes? Does changing survey responses indicate that people do not have well-formed opinions

Theory Nonseparable Preferences: What a person wants on one issue depends on what she gets on another issue Separable Preferences: What a person wants on every issue is independent of what they get on other issues

Measuring Nonseparable Preferences

Method Randomize the order of pairs of survey questions – For some issues, aggregate responses different across question order Each subject answers questions in order – Issue 1 then Issue 2 – Issue 2 then Issue 1

Method Impute what subject would have answered had they heard questions in different order For each question we then have Y i (if first) – Y i (if second) One of these will be imputed for each person since they cannot answer a question both first and second in the order First study to analyze individual differences in question orders, not simply aggregate differences

Conclusions Nonseparable preferences explain question order effects Political information level does not Response instability not due to uninformed respondents

Are Survey Experiments Externally Valid? JASON BARABAS and JENNIFER JERIT American Political Science Review 2010

Question Many survey experiments expose subjects to different information to show effect of on responses In a survey experiment, subjects are a “captive audience” that must pay attention Do the same information effects appear in the real world Compare survey experiments with natural experiments

Method Survey experiments give people to political information about immigration and medical care Pre-post survey also in field during change in medical insurance and immigration – Ask respondents which media sources they use Is the effect of information in the survey experiment as large as in the natural experiment?