1. Response bias People respond differently to how they believe People respond differently to how they believe Deliberate bias Deliberate bias Do you agree that abortion, the murder of innocent beings, should be outlawed? Do you agree that abortion, the murder of innocent beings, should be outlawed? Unintentional bias Unintentional bias Do you or do you not use drugs? Do you or do you not use drugs? People often want to please the interviewer People often want to please the interviewer Affected by sex, attire, race, behavior of interviewer Affected by sex, attire, race, behavior of interviewer Wording, Ordering, Complexity of Questions Wording, Ordering, Complexity of Questions

2. Wording the question Do you agree? Two questions from The New York Times (April, 1982) Do you agree? Two questions from The New York Times (April, 1982) (1) A freeze in nuclear weapons should be opposed because it would do nothing to reduce the danger of thousands of nuclear weapons already in place and would leave the Soviet Union in a position of nuclear superiority. (1) A freeze in nuclear weapons should be opposed because it would do nothing to reduce the danger of thousands of nuclear weapons already in place and would leave the Soviet Union in a position of nuclear superiority. (2) A freeze in nuclear weapons should be favored because it would begin a much-needed process to stop everyone in the world from building nuclear weapons now and reduce the possibility of nuclear war in the future. (2) A freeze in nuclear weapons should be favored because it would begin a much-needed process to stop everyone in the world from building nuclear weapons now and reduce the possibility of nuclear war in the future. Results: 58% agreed with (1). 56% agreed with (2), and 27% agreed with both! Results: 58% agreed with (1). 56% agreed with (2), and 27% agreed with both!

3. Open versus closed questions What do you think is the most important problem facing the country today? Which of the following do you think is the most important problem facing the country today---the energy shortage, the quality of public schools, legalized abortion, or pollution---or, if you prefer, you may name a different problem as most important. –From Problems in the use of survey questions to measure public opinion, Science, Volume 236 (1987)

4. Open versus closed questions Results of 171 responses to open question and 178 responses to closed question ProblemOpenClosed Energy0.0%5.6% Schools1.2%32.0% Abortion0.0%8.4% Pollution1.2%14.0% Others93.0%39.3% Dont know 4.7%0.6%

5. Another type of response bias Some Some people say that the 1975 Public Affairs Act should be repealed. Do you agree or disagree that it should be repealed. Washington Post, Post, Feb. 1995 Results: Results: For repeal: 24%, Against repeal: 19%, No opinion: 57% No No such thing thing as the Public Affairs Act!

6. Other forms of bias Selection bias Selection bias Method of selection produces unrepresentative sample Method of selection produces unrepresentative sample Shoppers at a mall: How do you feel about raising the sales tax? Shoppers at a mall: How do you feel about raising the sales tax? Non-response bias Non-response bias Telephone survey during work hours Telephone survey during work hours Voluntary response bias Voluntary response bias Should the drinking age be lowered to 18? Should the drinking age be lowered to 18?

7. You critique it Before 2000 election: What to do with large government surplus (1) Should the money be used for a tax cut, or should it be used to fund new government programs? (2) Should the money be used for a tax cut, or should it be spent on programs for education, the environment, health care, crime-fighting, and military defense? (1): 60% for tax cut; (2): 22% for tax cut

8. Telephone polling SRS assumes each household has known and equal chance of selection SRS assumes each household has known and equal chance of selection Not true Not true Multiple lines per household Multiple lines per household Households without telephones Households without telephones Number of adults per household Number of adults per household Some remedies Some remedies Pick from {oldest male, youngest male, oldest female, youngest female} Pick from {oldest male, youngest male, oldest female, youngest female} Random adult Random adult Last birthday Last birthday

9. Weighting the sample The sample first was weighted to take into account unequal probabilities of selection from sampling: Weighting accounts for the number of telephones going into the household, and household size. It then was weighted for age, gender, and education to take care of minor fluctuations in the sample, and align it with the findings of the 2000 Census of the adult population. It is assumed to be representative of all Minnesota households with telephones, within the margin of sampling error. – How the Poll was Conducted, Minneapolis Star Tribune, May 2, 2003

10. Weighting to adjust Suppose women are twice as likely to answer the phone as men Suppose women are twice as likely to answer the phone as men Weight survey results by multiplying womens responses by ½. Weight survey results by multiplying womens responses by ½. Responses: 150 men, 300 women Responses: 150 men, 300 women Count as: 150 men and 150 women Count as: 150 men and 150 women In practice, very complicated In practice, very complicated

11. Other sampling schemes Stratified sampling Goal: Random sample of 240 Carleton students Goal: Random sample of 240 Carleton students To insure discipline representation divide into strata according to population To insure discipline representation divide into strata according to population Arts and Literature 20% Arts and Literature 20% Humanities 15% Humanities 15% Social Sciences 30% Social Sciences 30% Mathematics and Natural Sciences 35% Mathematics and Natural Sciences 35% Choose 240 x.20 = 48 A and L students at random Choose 240 x.20 = 48 A and L students at random 240 x.15 = 36 Humanities 240 x.15 = 36 Humanities 240 x.30 = 72 Social science 240 x.30 = 72 Social science 240 x.35 = 84 Math and natural 240 x.35 = 84 Math and natural

12. Stratified sampling Advantages: Sample will be representative for the strata; Can gain precision of estimate Advantages: Sample will be representative for the strata; Can gain precision of estimate Disadvantages: Logistically difficult; must know about the population; May not be possible Disadvantages: Logistically difficult; must know about the population; May not be possible Note that a stratified sample is not a simple random sample Note that a stratified sample is not a simple random sample E.g., every possible group of 240 students is not equally likely to be selected E.g., every possible group of 240 students is not equally likely to be selected

13. Cluster sampling – an example Warehouse contains 10,000 window frames Warehouse contains 10,000 window frames stored on pallets Goal: Estimate how many frames have wood rot Goal: Estimate how many frames have wood rot Sample about 500 window frames Sample about 500 window frames Pallets numbered 1 to 400 Pallets numbered 1 to 400 Each pallet contains 20 to 30 window frames Each pallet contains 20 to 30 window frames Cluster sample Cluster sample Sample pallets, not windows. Choose, say 20. Sample pallets, not windows. Choose, say 20. Include in sample all the windows on each pallet Include in sample all the windows on each pallet

14. Cluster sampling Door-to-door surveys Door-to-door surveys City blocks are the clusters City blocks are the clusters Airlines get customer opinions Airlines get customer opinions Individual flights are the clusters Individual flights are the clusters Advantage: Much easier to implement depending on context Advantage: Much easier to implement depending on context Disadvantage: Greater sampling variability; less statistical accuracy Disadvantage: Greater sampling variability; less statistical accuracy

15. Multistage cluster sampling Stage 1: Regions (West, Midwest, Northeast, South) Demographic groupings Towns Stage 1: Regions (West, Midwest, Northeast, South) Demographic groupings Towns Stage 2: Towns Wards Stage 2: Towns Wards Stage 3: Wards Precincts Stage 3: Wards Precincts Stage 4: Precincts Households Stage 4: Precincts Households Individuals interviewed, but no discretion (e.g., speak to oldest woman at home 18 or older, or if no woman is at home, the youngest man 18 or older. Individuals interviewed, but no discretion (e.g., speak to oldest woman at home 18 or older, or if no woman is at home, the youngest man 18 or older. Offers advantages of quota sampling without disadvantage of selection bias Offers advantages of quota sampling without disadvantage of selection bias

16. Modern telephone surveys Multistage cluster design Multistage cluster design 4 time zones, 3 areas of population density = 12 strata 4 time zones, 3 areas of population density = 12 strata Random digit dialing (RDD) SRS of telephone numbers Random digit dialing (RDD) SRS of telephone numbers Telephone directory biased toward middle-class Telephone directory biased toward middle-class Computer excludes businesses Computer excludes businesses Those without phones creates small bias Those without phones creates small bias Interviewing done on evenings, weekends Interviewing done on evenings, weekends Will call back up to 3 times if no answer Will call back up to 3 times if no answer Refusal rate 20%; Cost 1/3 as much as personal interviews Refusal rate 20%; Cost 1/3 as much as personal interviews

17. Do you believe the poll/survey? Our graduates earn on average $60,000 per year Who carried out survey? Who carried out survey? What is the population? What is the population? How was sample selected? How was sample selected? How large was the sample? How large was the sample? What was the response rate? What was the response rate? How were subjects contacted? How were subjects contacted? When was the survey conducted? When was the survey conducted? What are the exact questions asked? What are the exact questions asked?