SESRI Workshop on Survey-based Experiments Session 2: How and Why We Design Experiments April 16, 2017 Doha, Qatar

Outline: Session 2 Features: why we use survey experiments Elements: key concepts Approaches: methods vs. substance

Why We Use Survey Experiments Combine benefits of population-based surveys with experiments External validity/generalizability (from population-based surveys) Internal validity/causal inference (from experimental design)

External Validity/ Generalizability Does the study allow me to make inferences about populations of interest? Key is representativeness Not all survey experiments embedded in representative survey samples, limits external validity Threats to external validity: unrealistic treatment

Internal Validity/Causal Inference Does the study allow me to conclude that the treatment caused the outcome? Key is randomization, ruling out other causes Threats to internal validity: failures of randomization, experimental effects, trend effects

Three Conditions for Establishing Causality 1. Covariation 2. Temporal order 3. Elimination of rival hypotheses/ alternative explanations

Elements Population-based surveys Randomization Treatment/independent variable Effect/outcome/dependent variable

Population-based Surveys “Strictly speaking, population-based survey experiments are more experiment than survey.” (Mutz 2011, p. 3) Survey does not need to be representative.

Population-based Surveys Target population: set of units/people about which inferences are to be made. Who is eligible and ineligible Representativeness Probability (random) vs non-probability samples Approaches Simple, stratified, clustered, post-stratification Challenges Need to give an example of Eligibility

Randomization AKA random assignment Each participant has an equal probability of being assigned to each group Ensures that groups are equivalent in expectation Challenges

Clicker Question 3 How do you know if the experimental design was effective? Treatment group and control group are same size Dependent variable has different values in treatment group and control group There is equivalence among covariates Subjects were randomly assigned to groups

Random Assignment Example: Selected Sample PID AGE 1 35 2 40 3 85 4 19 5 77 6 56 7 88 8 22 9 64 10 31

Random Assignment Example: Participant Age Method: Select odds and evens (from unsorted list) Method: Random numbers table Random Assignment 1 Treatment Control 35 19 64 22 77 31 85 40 88 56 Random Assignment 2 Treatment Control 31 19 35 22 40 56 64 77 85 88 Avg age: 69.8 Median: 77 Avg age: 33.6 Median: 31 Avg age: 51 Median: 40 Avg age: 52.4 Median: 56

Treatment/ Independent Variable In survey experiments, researcher controls assignment of the treatment. Treatment and control groups are identical in expectation pre-treatment, therefore any differences post-treatment are attributed to the treatment/independent variable. In analysis, no need to “control for” other causes/ mediating factors.

Effect/ Outcome/ Dependent Variable Dependent variable is the outcome to be explained. In a survey experiment, any statistically significant differences in the dependent variable between the treatment and control group is attributed to the treatment. May be differences of means, proportions, distributions, etc.

Approaches Methods-based experiments Substantive experiments

Methods-based Survey Experiments Seek to improve the survey process Test hypotheses about how changing aspects of surveys affects responses Challenge - what is the “true” response?

Substantive Survey Experiments Seek to test substantive/behavioral hypotheses Often lines between methods and substantive experiments are blurred