1. The Worst Social Statistic Ever
Source: Joel Best (2001), Damned Lies and Statistics

2. The quote in a dissertation prospectus
“Every year since 1950, the number of American children gunned down has doubled”.

3. Quote from the journal article that was the source: Same thing
“Every year since 1950, the number of American children gunned down has doubled”.

4. What???.. Think of the implications
For sake of argument let’s say that “the number of American children gunned down in 1950” was ONE.

5. Then… In 1951, there must have been two children gunned down
In 1952, there were four
In 1953, there were eight
And so on

6. And then… By 1960, the number would have been 1,024
By 1965 it would be 32,768
(even though the FBI only reported about 10,000 criminal homicides)
By 1970, the number would have surpassed 1 million

7. You see where we’re going…
By 1980, it would have been 1 billion (with a “B”)
More than 4 times the population of the US
By 1995, when the article was published, the annual number of victims would have been over 35 trillion (with a “T”)

8. Where did the author get the statistic?
Came from a 1994 report of the Children’s Defense Fund…
But, this is the quote…
“The number of American children killed each year by guns has doubled since 1950.”

9. Note the difference in wording
The CDF claimed that there twice as many deaths in 1994 as in 1950.
The author reworded the claim and created a very different meaning.

10. What about CDF’s claim? Where did the statistics come from?
Who counts child gunshot deaths and how?
Who is defined as “child”?
“Killed by guns”…does that mean suicides and accidents AND homicides?

11. There are some statistics that are bad from the start…
A Mutant Statistic
By the rewording, the author created a mutant statistic, one garbled almost beyond recognition
There are some statistics that are bad from the start…
Based on bad data or guesses

12. Bad or Mutant: Either way, they’re important
They can be used to stir up public outrage or fear
They can distort our understanding of our world
They can lead us to make poor policy choices

13. Mutant Statistics: Methods for Mangling Numbers
Generalization from bad definitions, bad measurement, bad samples
Transformation
150,000 people w/ anorexia became 150,000 deaths from anorexia
Crimes committed by “unknown” became committed “by strangers” in Supplemental Homicide Report

14. Mutant Statistics: Methods for Mangling Numbers, p2
Confusion: garbling complex statistics
Compound errors
Number repeated over and over again giving it more credibility
Example: Kinsey reports (pp.88-89); Hite Report (self-selected sample)

15. Hite Report: A Nationwide Study of Female Sexuality, 1976
“Interesting” sampling & data-gathering method
Hite paid to have surveys printed in a number of national publications such as Ms. Magazine, the Village Voice, and Modern Bride …..
…..and invited readers to answer the questions (all requiring mini-essays)…..
…..and mail in the surveys

16. Hite Report(s): Findings
70% of responding women who had been married more than five years reported having an affair
76 % of them claimed not to have feelings of guilt about their infidelity
More than 95% of women surveyed for Hite's third book claimed to have suffered emotional and psychological harassment" from their men
98% replied that they desired "basic changes" in their relationships with husbands or lovers
At odds with studies conducted by other researchers
But, Hite generalized her findings to entire female U.S. population

17. Roots of Mutant Statistics
Innumeracy
Don’t understand statistics…
Make honest errors
Manipulation
Deliberate attempts to turn statistical information to particular uses

18. Characteristics of good statistics
Based on more than just guessing
Based on clear, reasonable definitions
Based on clear reasonable measures
Based on good samples

19. Sources of Bad Statistics
Guessing
Defining
Measuring
Sampling

20. 1. Guessing
Activists’ estimates: people who create the statistic have a stake in it.
Dark figure—that proportion of crime, hunger, abuse, etc---that goes unreported
Number laundering---number takes on life of its own
Its origin as someone’s best guess becomes lost and it is quoted widely as if it were accurate---ex: homelessness
Scott Adams: “Reporters are faced with the daily choice of painstakingly researching stories or writing whatever people tell them. Both approaches pay the same.” (p. 35)
Examples: Delaware gun violence study

21. 2. Defining What is the nature of the problem?
Whenever examples substitute for definitions there is great risk that our understanding of the nature of the problem will be distorted
Broad definitions to encompass as many cases as possible
Example: sexual violence
False positives vs false negatives
Activists more bothered by false negatives

22. 3. Measuring Social life is complex and measurement must be precise
Measurement involves choices
Wording of questions affects measurement
Advocacy research…
Abortion, Violent crime, UFO examples

23. Measuring Crime
Research group* says that 47% of public in Northeast (9 states) have been the VICTIM of a VIOLENT crime
What???...seems awfully high
*Schulman, Ronca and Bucuvalas in a study done for the Council of State Governments/Eastern Regional Conference, 1999

24. Measuring Crime, p.2
To get that finding, researchers combined the answers to 6 questions on the survey regarding victimization
Each answer was a Yes or NO to the following questions…

25. Measuring Crime, p.3 Violence Questions
Has anyone ever robbed or tried to rob you with a gun, knife or some other weapon?
Aside from robbery, has anyone ever physically attacked you with a gun, knife or some other weapon?
Has anyone ever physically attacked you with a weapon, but with intent to seriously harm you?

26. Measuring Crime, p.4 Violence Questions
Has anyone ever threatened to attack or seriously harm you?
Has anyone ever forced you or tried to force you to have sex against your will?
Aside from what you have already told me, have you ever been the victim of any other violence or threat of violence , including injury by a drunken driver, regardless of whether you feel it was a crime?

27. Measuring Crime, p.5
A “yes” response to any ONE of the previous questions qualified the respondent as a victim of violent crime
The “catch-all” last question virtually assured the finding of a high percentage of respondents as victims

28. Which “measure” do you pick
Which “measure” do you pick? Percentage of Americans Favoring Legal Abortions under Different Circumstances, 1996
1. If the woman’s own health is seriously endangered by pregnancy 92
2. If she became pregnant as a result of rape 84
3. If there is a strong chance of serious defect in the baby 82
4. If she is married and does not want any more children 47
5. If the family is very low income and can’t afford any more children
6. If she is not married and does not want to marry the man 45
7. If the woman wants it for any reason

29. Measuring UFO contacts: Depends on how you ask
Research organization reported that 2% of Americans (about 4 million people) had been abducted by UFOs
How did researchers arrive at that figure?
Did they ask: “Have you ever been abducted by a UFO?”
Didn’t ask…they argued that folks wouldn’t know it if they had been abducted

30. Measuring UFO contact, p2
Devised five “symptoms” of UFO contact
For example, “Waking up paralyzed with a sense of a strange person or presence or something else in the room.”
Concluded that anyone who reported four or more symptoms probably had been abducted
2 percent of survey respondents fell into this group…ergo, 2 percent of population had been abducted.

31. Measuring UFO contact, p3
Points to measurement choices
If the researchers had decided that only one or two symptoms were necessary for UFO contact, the proportion of abductees would have been higher
If they had required all five symptoms, the percentage would have been lower.
You can get any percentage you want by changing the measurement choices

32. 4. Sampling
Representativeness of sample is more important than sample size
Often, random selection is compromised by time, money, ideology.

33. Characteristics of good statistics
Based on:
More than just guessing
Clear, reasonable definitions
Clear, reasonable measures
Good samples

34. 1. more than guessing Watch for signs of guessing…
Do people offering statistic have a bias?
Is the statistic a BIG, ROUND number?
Tell a lie…tell a big one
Does it describe some unfamiliar social problem…a dark figure?

35. 2. Clear, reasonable definitions
Definitions of subject must be public
An example---particularly dramatic or disturbing or horror story or worst case---is NOT a definition

36. 3. Clear, reasonable measures
How was the social problem measured?
The measurement must be reasonable
For example, could ask the question in biased way…Public Agenda experience

37. 4. Based on good samples
Ideally based on random sample of the larger population to which the generalization will be made
Watch out for statistics that are based on small, non-random convenience samples
Food supplement research UD

38. Moral Edward Tufte, story about the spider and the fly
“There is no safety in numbers…or anything else.”