Designing and Using a Survey February 7, 2011

Objectives By the end of this meeting, participants should be able to: Explain how surveys accomplish the goals of: measuring attitudes, measuring change over time, making group comparisons, and analyzing the causes of behavior.

Goals of Surveys Measure the frequency of attitudes, beliefs and behaviors Measure change over time Examine differences between groups (race, class, gender, etc.) Analyze causes of attitudes, beliefs, and behaviors

Measuring Behavior Time can be awkward for respondents Forward telescoping Backward telescoping Sensitive subjects Bogus pipeline technique (ethical concerns?) Random lists Randomized response technique (some sensitive, some not) Importance of others ’ beliefs

Predicting Behavior Election Predictions Problems with third parties Problems with really close elections Problem with social desirability for turnout Undecided Sensitive topics

2008 Presidential Example- NC Final Results Obama2,142, % McCain2,128, % Others39, % Total4,310,851 Source: Office of the Clerk of the US House of Representatives

How did the individual polls do? FOX News/RasmussenMcCainMofE Reuters/ZogbyMcCainMofE American Research GroupObamaMofE SurveyUSAMcCainMofE Mason-DixonMcCainMofE Research 2000ObamaMofE Politico/InsiderAdvantageTieMofE

Recall: Margin of Error So for example, FOX/Rasmussen ’ s final poll for North Carolina showed 50% for McCain and 49% for Obama Sample: 1000 likely voters Margin of error = ±t {p(1−p)/(n−1)} 1/2 {1 − f} 1/2 But since the population being sampled is large we can ignore the final {1 − f} 1/2 ±1.96 {50(50)/( )} 1/2 = ±3.1% In other words, Rasmussen/FOX was 95% confident that the final vote for McCain would be between 46.9% and 53.1%. Right on the mark, pretty much. Yet, if they had to make a prediction of the winner, they ’ d guess wrong. (Some polls had MO wrong, too.)

Measuring Attitudes One of the most common goals of surveys Need to be concerned about measuring non-attitudes (Converse) Need to consider attitude strength, respondents may not care equally about all issues (environment, gun control, etc.) How do they feel about the issue personally? How knowledgeable are they about this issue? How certain are they of their opinion? How much thought they have given to the issue? Should you offer counterarguments?

Measuring attitudes d)Attitudes especially weakly held attitudes will change over time (first impression versus considered opinions) e)Need to watch disagreements over the meaning of words Example: ideology and Stimson ’ s work f)Need to watch for changing frames Verbal (example: welfare, next page) Contextual (example: sexual harassment)

Example of Verbal Differences- GSS

Measuring Change over Time Measurement of personal attitude change is frequently unreliable Comparisons of cross sectional surveys are more common but pose their own issues Similar interviewing methods? Similar sample sizes? Similar question wordings? Does the change exceed sampling variation?

Measuring Change over Time c)Frequently can be atheoretical d)Can only measure gross change, not change at the individual level e)Insta-polls can allow for immediate reactions to stimuli but suffers worries about representativeness and long term impact

Measuring Change over Time Preferred method: panel studies Repeated studies of the same individuals over time Limitations Long term studies are prohibitively expensive Atrophy of the original sample (and worries about those that remain) Errors in interviewing

Sub-group Comparisons Researchers frequently want to compare the attitudes of various sub-groups of the population One difficulty is the smaller and less accessible groups may be harder to randomly sample Double sample or over sampling Pyramiding or combining multiple surveys

What Leads to Change in Attitudes over Time? Surveys generally struggle with this type of question Most people ’ s answers for why their opinions changed are post-hoc rationalizations Individuals may not be sure of their reasons for opinion change Similar stimulus may affect people differently

For February 9: Work on research proposals, you wi ll have time for group work next meeting. For February 11: Download the program R from and bring your laptop to class. You also may want to print-out “Getting Our Feet Wet with R” from the website. For February 14: – Research proposals due for each group. – Read WKB Chapter 8 – On p. 189 answer either question 1, 2, or 3. – Construct a causal model relevant for YOUR research question. (Turn-in the C.8 items individually.)