Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 1 Statistical Reasoning 5

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 2 Unit 5B Should You Believe a Statistical Study?

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 3 Most statistical research is carried out with integrity and care. Nevertheless, statistical research is sufficiently complex that bias can arise in many different ways. There are eight guidelines that can help you answer the question “Should I believe a statistical study?” Evaluating a Statistical Study

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 4 Guideline 1 Get a Big Picture View of the Study. Try to answer the following questions. What was the goal of the study? What was the population under study? Was the population clearly and appropriately defined? What type of study was used? Was the type appropriate for the goal?

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 5 Guideline 2 Consider the source. Statistical studies are supposed to be subjective, but the people who carry them out and fund them may be biased.

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 6 Guideline 3 Look for bias in the sample. Selection bias occurs whenever researchers select their sample in a way that tends to make it unrepresentative of the population. Participation bias occurs primarily with surveys and polls; it arises whenever people choose whether to participate. Because people who feel strongly about an issue are more likely to participate, their opinions may not represent the larger population.

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 7 Guideline 4 Look for problems in defining or measuring the variables of interest. A variable is any item or quantity that can vary or take on different values. The variables of interest in a statistical study are the items or quantities that the study seeks to measure.

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 8 Example A commonly quoted statistic is that law enforcement authorities succeed in stopping only about 10% to 20% of the illegal drugs entering the United States. Should you believe this statistic? Solution There are essentially two variables in the study: quantity of illegal drugs intercepted and quantity of illegal drugs NOT intercepted. It should be relatively easy to measure the quantity of illegal drugs that law enforcement officials intercept.

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 9 Example (cont) However, because the drugs are illegal, it’s unlikely that anyone is reporting the quantity of drugs that are not intercepted. How, then, can anyone know that the intercepted drugs are 10% to 20% of the total? In a New York Times analysis, a police officer was quoted as saying that his colleagues refer to this type of statistic as “PFA,” for “pulled from the air.”

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 10 Guideline 5 Beware of confounding variables. Variables that are not intended to be part of the study can sometimes make it difficult to interpret the results properly. Such variables are often called confounding variables, because they confound (confuse) a study’s results.

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 11 Guideline 6 Consider the setting and wording in surveys. Even when a survey is conducted with proper sampling and with clearly defined terms and questions, it is important to watch out for problems in the setting or wording that might produce inaccurate or dishonest responses.

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 12 Example The Republican National Committee commissioned a poll to find out whether Americans supported their proposed tax cuts. Asked “Do you favor a tax cut?” a large majority answered yes. Should we conclude that Americans supported the proposal?

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 13 Example (cont) Solution A question like “Do you favor a tax cut?” is biased because it does not give other options (much like the fallacy of limited choice discussed in Unit 1A). In fact, other polls conducted at the same time showed a similarly large majority expressing great concern about federal deficits. Indeed, support for the tax cuts was far lower when the question was asked by independent organizations in the form “Would you favor a tax cut even if it increased the federal deficit?

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 14 Guideline 7 Check that results are presented fairly. The study may be misrepresented in graphs or concluding statements. Researchers may misinterpret the results or jump to conclusions not supported by the results.

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 15 Example The school board in Boulder, Colorado, created a hubbub when it announced that 28% of Boulder school children were reading “below grade level” and hence concluded that methods of teaching reading needed to be changed. The announcement was based on reading tests on which 28% of Boulder school children scored below the national average for their grade. Do these data support the board’s conclusion?

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 16 Example (cont) Solution The fact that 28% of Boulder children scored below the national average for their grade implies that 72% scored at or above the national average. Therefore, the school board’s ominous statement about students reading “below grade level” makes sense only if “grade level” means the national average score for a particular grade. This interpretation of “grade level” is curious because it means that half the students in the nation are always below grade level— no matter how high the scores. The conclusion that teaching methods needed to be changed was not justified by these data.

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 17 Guideline 8 Stand back and consider the conclusions. Ask yourself the following questions. Did the study achieve its goals? Do the conclusions make sense? Can you rule out alternative explanations for the results? If the conclusions do make sense, do they have any practical significance?

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 18 Example An experiment is conducted in which the weight losses of people who try a new “Fast Diet Supplement” are compared to the weight losses of a control group of people who try to lose weight in other ways. After eight weeks, the results show that the treatment group lost an average of 1/2 pound more than the control group. Assuming that it has no dangerous side effects, does this study suggest that the Fast Diet Supplement is a good treatment for people wanting to lose weight?

Copyright © 2015, 2011, 2008 Pearson Education, Inc. Chapter 5, Unit B, Slide 19 Example (cont) Solution Compared to the average person’s body weight, the difference of 1/2 pound hardly matters at all. So even if the study is flawless, the results don’t seem to have much practical significance.