1. Chapter 41 Sample Surveys in the Real World

2. Chapter 42 Thought Question 1 (from Seeing Through Statistics, 2nd Edition, by Jessica M. Utts, p. 14) Nicotine patches are patches attached to the arm that dispense nicotine into the blood of someone who is trying to quit smoking. Suppose you were to read about a study showing that nicotine patches were twice as effective in getting people to quit smoking as “control” patches (made to look like the real thing). Further, suppose you are a smoker trying to quit. What questions would you want answered about the study before you decided whether or not to try the patches yourself?

3. Chapter 43 Thought Question 2 When surveying students on their opinions of their school’s athletics programs, do you think it matters who conducts the interviews? Explain your answer with an example.

4. Chapter 44 Thought Question 3 A survey on poverty and welfare included the following question, “Do you agree with the popular notion that government policy should attempt to assist those individuals who have had the misfortune to end up living in poverty by providing them with much needed financial assistance until they can get back on their feet?” Based on the wording, do you think the author of this question was looking for support or opposition to welfare programs? Explain.

5. Chapter 45 Questions to ask about a study (determining the soundness of statistical studies) 1. Who funded the study and who conducted the study [are they related - any problems with conflict of interest]? 2. Who were the individuals or objects studied, and how were they selected [population, sampling method, sample size, response rate, margin of error]? 3. What was the setting in which the measurements were taken [time, location, method of contact]? 4. What was the exact nature of the measurements made or questions asked [are there problems with the measurements used, question ordering, question wording]? 5. Were there any other possible differences in the groups being compared [any confounding/lurking variables]? 6. What was the magnitude of any claimed effects or differences [are the numbers correct, statistically significant differences, practical differences]?

6. Chapter 46 Case Study Questions to ask Nicotine Patches for Quitting Smoking from Thought Question 1

7. Chapter 47 Questions to ask Nicotine Patches for Quitting Smoking 1. Who funded the study, and who conducted the study (and are they related)? Funding Source: Company producing nicotine patch? Independent research organization? Researchers: People from the funding agency, possibly with direct interest in the results? Hired persons who do not know the goals of the research?

8. Chapter 48 Questions to ask 2. Who were the individuals or objects studied, and how were they selected? What is the population of interest? Were all subjects trying to quit smoking? Age, gender, previous smoking behavior,...? Volunteers? Recruited by researchers? 3. What was the setting in which the measurements were taken? Probably was in natural environment. How were subjects contacted and measured? Any measurements done in a lab? Nicotine Patches for Quitting Smoking

9. Chapter 49 Questions to ask 4. What was the exact nature of the measurements made or questions asked? How was ‘quitting’ measured (for how long)? Group assignments were probably recorded. 5. Were there any other differences in the groups being compared? Could nicotine and ‘control’ groups be different in regards to age, gender, smoking behavior, etc.? Were subjects randomly assigned to groups? Nicotine Patches for Quitting Smoking

10. Chapter 410 Questions to ask 6. What was the magnitude of any claimed effects or differences? Nicotine patches were “twice as effective” in getting people to quit as control patches. What are the actual percentages that quit?.01% &.02% ? 30% & 60% ? others ? Nicotine Patches for Quitting Smoking

11. Chapter 411 Brooks Shoes Brings Flawed Study to Court reported in Statistical Reasoning in Law and Public Policy, Vol. 2, pp. 517-520 Case Study Questions to ask

12. Chapter 412 Questions to ask 1. Who funded the study, and who conducted the study (and are they related)? Funding Source: Brooks shoe company. Brooks’ lawyer instrumental in designing survey. Researchers: ‘Interviewers’ – employees of the company? They were inadequately trained in how to conduct an unbiased survey. Brooks Shoes

13. Chapter 413 Questions to ask 2. Who were the individuals or objects studied, and how were they selected? 121 spectators & participants at 3 track meets. Not selected to be ‘representative’ of the general public in the area (e.g., they were more educated). 3. What was the setting in which the measurements were taken? Track meets, where people are more likely to be familiar with athletic shoes. Brooks Shoes

14. Chapter 414 Questions to ask 4. What was the exact nature of the measurements made or questions asked? Identifying Brooks shoe. Also, “how long have you known about Brooks Running Shoes?” 5. Were there any other differences in the groups being compared? Probably not a problem here. Brooks Shoes

15. Chapter 415 Questions to ask 6. What was the magnitude of any claimed effects or differences? 71% recognized a Brooks shoe [33% of these due to the ‘V’ design] 39% thought Suave shoe was Brooks shoe [48% of these due to the ‘V’ design] Brooks Shoes Suave conducted a proper survey of 404 respondents and found that only 2.7% recognized a Brooks shoe from the ‘V’!

16. Chapter 416 Questions to ask about a study “University Students are Healthier than you Think” Spring 2002 random undergraduate classroom survey of n=810 students was conducted by the Office of Health Promotion within the University Student Health Services, Division of Student Affairs. Statistics from this survey led to the following conclusions: - most students (67%) have 0-4 drinks when they go out - most (69%) have had 0-1 sex partners in the past year - most (76%) either don’t drink, or use designated drivers if they do What questions should you ask to help you assess the credibility of these results?

17. Chapter 417 Errors in Sample Surveys u Sampling errors u Random sampling errors –measured by margin of error u Nonsampling errors

18. Chapter 418 Sampling Errors u Difficulties –Using the wrong sampling frame (next slide) u Disasters –Using voluntary response (volunteer sample) –Using a convenience or haphazard sample v cannot extend results to the population of interest (need a broad cross-section of the population)

19. Chapter 419 Using the Wrong Sampling Frame Including some units not in the population. Undercoverage: Excluding some units in the population.

20. Chapter 420 Convenience Sample u Brooks Shoes study: –71% recognized a Brooks shoe u Properly selected sample by Suave Shoe Corp. –2.7% recognized a Brooks shoe

21. Chapter 421 Case Study Can Results be Extended? Meditation and Aging (Noetic Sciences Review, Summer 1993, p. 28) u Haphazard sample: u Meditators were recruited from a university in Iowa that specializes in teaching meditation u Non-meditators were recruited from a different setting in New York City

22. Chapter 422 u Difficulties –Processing errors (data entry, calculations) –Wording of questions / Response error u Disasters –Nonresponse (cannot contact subjects or they do not respond) Nonsampling Errors

23. Chapter 423 Concerns when Asking Survey Questions u Deliberate bias u Unintentional bias u Desire to please u Asking the uninformed u Unnecessary complexity u Ordering of questions u Confidentiality and anonymity

24. Chapter 424 Deliberate Bias u “If you found a wallet with $20 in it, would you return the money?” u “If you found a wallet with $20 in it, would you do the right thing and return the money?”

25. Chapter 425 Unintentional Bias u “I have taught several students over the past few years.” –How many students do you think I have taught? –How many years am I referring to? u “Over the past few days, how many servings of fruit have you eaten?” –How many days are you considering? –What constitutes a serving?

26. Chapter 426 Desire to Please u “Is your instructor doing a good job presenting the course material in a clear and interesting way?” v Yes v No

27. Chapter 427 u A 1978 poll done in Cincinnati asked people whether they “favored or opposed repealing the 1975 Public Affairs Act.” –There was no such act! –About one third of those asked expressed an opinion about it. Washington Post National Weekly Edition (April 10-16, 1995, p. 36) Asking the Uninformed: Case Study

28. Chapter 428 Unnecessary Complexity u “Do you sometimes find that you have arguments with your family members and co-workers?” –Arguments with family members –Arguments with co-workers –“sometimes find” (vague or unclear)

29. Chapter 429 Ordering of Questions u “How often do you normally go out on a date? about ___ times a month.” u “How happy are you with life in general.” –Strong association between these questions. –If the ordering is reversed, then there would be no strong association between these questions

30. Chapter 430 Confidentiality and Anonymity u Confidential answer –respondent is known, but the information is a secret –facilitates follow-up studies u Anonymous answer –the respondent is not known, or cannot be linked to his/her response –usually yields more truthful answers

31. Chapter 431 Probability Sampling Plans u Simple random sampling (SRS) u Stratified random sampling u Cluster sampling u Systematic sampling u Random digit dialing u Multistage sampling

32. Chapter 432 Stratified Random Sample u first divide the population into groups of similar individuals, called strata u second, choose a separate simple random sample in each stratum u third, combine these simple random samples to form the full sample –if only certain strata are (randomly) chosen to be used, and all subjects in these strata make up the sample, then we have a cluster sample

33. Chapter 433 Systematic Sample u randomly select a member of the sampling frame for the sample u using a set procedure or rule, select the rest of the individuals for the sample –for example, randomly select an individual from the sampling frame, and then select every 25 th member of the sampling frame to be in the sample

34. Chapter 434 Multistage Sample u divide the population of interest into groups u randomly select some of those groups u divide the resulting collection of individuals into smaller groups u randomly select some of those groups u continue dividing the resulting collection of individuals into groups and randomly selecting some of those groups until you can simply list all of the resulting individuals and randomly select n of them for your sample

35. Chapter 435 Multistage Sample u Example: Selecting 1500 registered U.S. voters [Use multistage sampling since we cannot obtain a sampling frame (list) of all registered U.S. voters.] –randomly select five U.S. states –obtain a list of all counties/cities in those states –randomly select 20 of those counties/cities –obtain a list of all registered voters in those 20 counties/cities –randomly select 1500 voters from that list

36. Chapter 436 Key Concepts u Ask questions when you read about studies. u Sampling & Nonsampling Errors –Concerns when asking survey questions u Stratified Sampling, Cluster Sampling, Systematic Sampling, Multistage Sampling Continued… 

37. Chapter 437 In summary… Questions to ask before you believe a study/poll 1. Who funded the study and who conducted the study [are they related - any problems with conflict of interest]? 2. Who were the individuals or objects studied, and how were they selected [population, sampling method, sample size, response rate, margin of error]? 3. What was the setting in which the measurements were taken [time, location, method of contact]? 4. What was the exact nature of the measurements made or questions asked [are there problems with the measurements used, question ordering, question wording]? 5. Were there any other possible differences in the groups being compared [any confounding/lurking variables]? 6. What was the magnitude of any claimed effects or differences [are the numbers correct, statistically significant differences, practical differences]?