Measurements, Mistakes and Misunderstandings in Sample Surveys Lecture 1

Goals Understand the difficulty of precisely defining and measuring something. Realize bias can enter surveys many ways, even with a good sample. Compare open and closed questions. Understand why random sampling works

Simple Measures......don’t exist!

Measurement Variability Variable measurements include unpredictable errors or discrepancies that aren’t easily explained. Natural variability is the result of the fact that individuals and other things are different.

Reasons for variable measures Measurement error Natural variability between individuals Natural variability over time in a single individual Statistics are tools to help us work with measurements that vary

Some Pitfalls in Studies Deliberate bias

Deliberate Bias? If you found a wallet with $20, would you: “Keep it?” “Do the honest thing and return it?”

Deliberate Bias? If you found a wallet with $20, would you: “Keep it?” (23% would keep it) “Do the honest thing and return it?” (13% would keep it)

Unintentional Bias? “Do you use drugs?” “Are you religious?”

Desire to Please? People routinely say they have voted when they actually haven’t, that they don’t smoke when they do, and that they aren’t prejudiced. One study six months after an election: 96% of actual voters said they voted. 40% of non-voters said they voted.

Asking the uninformed? Washington Post poll : “Some people say the 1975 Public Affairs Act should be repealed. Do you agree or disagree that it should be repealed?” 24% said yes 19% said no rest had no opinion

Asking the uninformed? Later Washington Post poll: “President Clinton says the 1975 Public Affairs Act should be repealed. Do you agree or disagree that it should be repealed?” 36% of Democrats agreed 16% of Republicans agreed rest had no opinion

Unnecessary Complexity? “Do you support our soldiers in Iraq so that terrorists won’t strike the U.S. again?”

Unnecessary Complexity? Roper 1993: “Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?” 22% said it was possible and 12% more said they didn’t know

Unnecessary Complexity? Gallup 1994: “Do you doubt that the Holocaust actually happened?” Only 9% said they had doubts Only about 2% said the Holocaust “definitely” or “probably” didn’t happen

Question Order “About how many times a month do you normally go out on a date?” “How happy are you with life in general?”

Closed vs. Open questions Closed: Respondent given list of choices Open: Respondent answers in own words Problem with closed format: Choices may not reflect respondent’s ideas Problem with open format: Difficult to summarize

How to Get A Good Sample

Common Research Methods Randomized experiments: Measure deliberate manipulation of the environment Observational studies: Measure the differences that occur naturally Meta-analyses: Quantitative review of multiple studies Case Study: Descriptive in-depth examination of one or a few individuals

Common Research Methods Experiments Observational Studies Meta-Analyses Case Studies Sample Surveys

Why Sample? A properly-selected random sample will mirror the characteristics of the population from which it was taken. The accuracy (margin of sampling error) depends on size of the sample -- not the size of the population! A sample survey is much cheaper and faster than a census. Sometimes impossible to do a census. Sample can be more accurate.

Margin of Error 95% of the time, a random sample’s characteristics will differ from the population’s by no more than about where N= sample size (Yes, memorize this formula.)

Margin of Error Example Imagine that 70% of a random sample of 400 UNL students say professors should wear propeller beanies while lecturing. About what percentage of all students think that?

Calculating Margin of Error Square root of 400 = 20 1/20 = 0.05 = 5 percentage points 70% plus or minus 5 points 70-5 = 65 and 70+5 = 75 Therefore, the “real” percentage almost certainly is within +/- 5 percentage points, or between 65 and 75 percent.

Margin of Error Example Imagine Estádio da Luz filled with 25,000 men and 25,000 women. Pick at random: 100 people -- Get 40.0%-60.0% men 200 people -- Get 43.0%-57.0% men 400 people -- Get 45.0%-55.0% men 900 people -- Get 46.7%-53.3% men 1,600 people -- Get 47.5%-52.5% men

Two Important Concepts about Error Margin The larger the sample, the smaller the margin of sampling error. The size of the population being surveyed doesn’t matter.* *Unless the sample is a significant fraction of the population.

Sampling realities Bigger sample means more cost (money and/or time) Diminishing return on error margin improvement as sample increases. Sample needs only to be large enough to give a reasonable answer. Sampling error affects subsamples, too.

Simple Random Sampling Everyone should have the same chance of being sampled. All that’s needed is: A list of the population Random numbers (or some other physical method of selection, like drawing names from a hat.)

Systematic Sampling Pick a random starting place in a list, then survey every nth person on the list... * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Random Digit Dialing Generate telephone numbers by randomly selecting a known exchange, then four randomly-chosen digits Advantages of RDD? Disadvantages of RDD?

Difficulties in Sampling The major difficulties in doing a survey are getting a good sample and getting them to respond. Reasons include: Using the wrong “universe”. Not reaching the individuals selected. Getting low response or getting a volunteer response.

Disasters in Sampling Some kinds of samples are so bad they make the so-called “survey” worthless: Volunteer (self-selected) samples Mail-in surveys Phone-in surveys Convenience samples, such as so-called “man in the street” surveys

Literary Digest Poll In 1936, the Literary Digest magazine sent out 10 million surveys and from the responses predicted Alf Landon would beat FDR by a 60 to 40 margin. George Gallup surveyed 50,000 people: He correctly predicted FDR would win And correctly predicted how wrong the Literary Digest poll would be

Perguntas?