Survey Experiments: Illuminating Theoretical and Methodological Controversies Harold Clarke EPPS, University of Texas at Dallas

Utility of Survey Experiments Survey experiments – increasingly popular over past decade Sold as way to study causal processes that cannot be investigated with traditional cross-sectional & panel surveys Randomization as cure for omitted variable biases & endogeneity problems that bedevil causal inference in observational studies Survey experiments – external validity - sophomores in labs are not representative! Other problems – truncated variance, small N’s, functional form forgotten by ANOVA, model specification

Survey Experiment Vehicles In-person probability surveys – ANES, BES, GSS Mail-back surveys RDD telephone surveys – impact of TESS – NSF funds Internet surveys – general populations - BES Continuous Monitoring Survey (CMS), BES Rolling Campaign Panel Survey (RCPS), now ANES Internet surveys – specific populations – Survey Monkey, Qualtrics -> e.g., students, employees, patients, party members Amazon Mechanical Turk too!

Internet Survey Experiments Internet methodology – exciting possibilities Huge N’s – massive statistical power Inexpensive – vastly expand research opportunities – e.g., CMS Technical tools – video, audio, feedback to respondents, intricate randomizations

Theory & Method Often think of experiments as theory or, at least, hypothesis testing Hypotheses of interest may concern measurement of key concepts Survey measures may be affected by context in which questions are asked Internal – question ordering effects External – political/economic conditions Gaines, Kuklinski & Quirk (2007) discuss the latter in terms of survey experiments, but it is of general interest Use survey experiments to investigate how external context affects survey responses or relationships among variables – cross-level interactions

External Context Survey Experiment: Inglehart and Post-Materialism Testing the impact of macro-economic context on responses to Inglehart’s Materialism-Postmaterialism question battery in the Euro-Barometer surveys Does Inglehart’s battery really measure what he says it does? Or, is it contaminated by changing economic context Why does this matter?

Inglehart’s Value Shift Thesis & His Values Battery Inglehart famously argued there has been a “value shift” from materialist to postmaterialism driven by rising affluence in post-war Western democracies. Everybody jumped on this bandwagon! Four-Item Values Battery administered in bi-annual Euro- Barometer surveys Items: Fight Rising Prices Maintain Order in the Nation Give People More Say in Government Protect Freedom of Speech Respondents asked to select the most important, then select the next most important of the 3 remaining items.

Figure 1. Why The Battery Might Be Affected by Macro-Economic Context

Figure 2. Germany – Canonical Case

Values Battery Survey Experiment Clarke’s Chance Encounter at Zuma Dinner With Max Kaase What are you working on Harold? Studying how Macro-Economic Conditions Affect Responses to Inglehart’s Values Battery Nice aggregate results, but need to nail it at the individual level Survey experiment! Parallel work being done by Kaase in Germany Let’s work together!

The Experiment National Samples of Canadian & German Electorates Split-Sample Design Half-One gets traditional Values Battery; Half- Two gets Values Battery with Unemployment item substituted for Inflation Item At end of Canadian survey, ask other battery – Do people change their value classification in a 15-minute survey?! Results?

Figure 3. Alternative Values Batteries, Responses - Canada

Figure 4. Materialists & Post-Materialists Rival Values Batteries - Canada

Results & Implications Same results in Germany! Significant portion of the hypothesized value shift an artifact of interaction between his measure and changing macroeconomic context Example of Gaines’ et al. caveat in their Political Analysis article Simple experiments can have big payoffs

Three Other Examples of Survey Experiments The Dynamics of Partisanship – Does party identification have an individual dynamic? One aspect of controversy concerns question wording – A task for survey experiments! Does Mode Matter? - Do Traditional In-Person and Internet Election Studies Tell the Same Story? The Endogeneity of Ideal Points? – Leveraging the Power of the Internet With a Feedback-to- Respondent Experiment to determine if “De Gustibus” is a viable psephological anthropology.

Dynamics of Partisanship Longstanding Controversy! Campbell et al. v. Fiorina et al. Green et al. In Canada, Richard Johnston & Andre Blais said if only we changed the “or what?” in first ANES party id question to “or none of these” we would find: 1. fewer identifiers 2. more stable identifiers

Survey Experiment on Stability of Party Identification in Canada Split Half Samples in Political Support in Canada Panel Surveys One Half Gets Traditional Wording; Other Half Gets Revised Wording Results: Slightly fewer identifiers with revised wording Rates of partisan stability almost exactly the same Lots of partisan instability in both versions Survey experiment results reinforce MMLC analyses of US, UK, Canadian and German multi-wave panel data Clarke & McCutcheon POQ 2009 and Stegmueller, Neundorf and Scotto, POQ 2011.

Figure 8. Dynamics of Party Identification, PSC Panel: Stable Identifiers Traditional and Revised Question Wordings

Does Mode Matter? Are Ideal Points Endogenous? Mode – see modecomp1.ppt Ideal Points – see endogjepoprev.pdf

Conclusions I Survey experiments are useful tools! Survey experiments can address big theoretical & methodological controversies Internet surveys put survey experiments “in reach” for many researchers CCAP, CCES, CMS – example vehicles Qualtrics – sophisticated platform – many universities subscribe – no cost

Conclusions II Survey experiments not a panacea Example of ongoing controversies over implications of endogenity of economic evaluations in party support and voting models Never resolve controversy with traditional cross- sectional or panel survey designs! Survey experiments very unlikely to simulate real economic conditions Way forward - use CMS monthly surveys & VEC modeling – see Snark Paper Data Driven Solutions Not Necessarily Expensive