Statistical Fallacies Catastrophes & Contributions, Statistical Literacy for ManagersStatLit for Managers StatLit for Managers Analyzing Numbers in the News 2013 1 March 20132013 15 May 2008 Statistical Fallacies Catastrophes & Contributions, [Abstract 3214116] Milo Schield, Augsburg College Member: International Statistical Institute US Rep: International Statistical Literacy Project Director, W. M. Keck Statistical Literacy Project VP. National Numeracy Network August 1, 2016 www.StatLit.org/pdf/2016-Schield-ASA-Slides.pdf First big idea: Statistical educators at JSM are an extremely biased sample compared to their student abilities and majors. 2008SchieldNNN6up.pdf 2013Schield-MBAA www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 1 1 1

Core Concepts in Intro Stats Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 2 Core Concepts in Intro Stats McKenzie (2004): Survey of Educators Goodall@RSS (2007) Big Ideas in Statistics Garfield & Ben Zvi (2008): Big Ideas of Statistics Gould-Miller-Peck (2012). Five Big Ideas Blitzstein@Harvard (2013): 10 Big Ideas Stat110 Stigler (2016): Seven pillars of statistical wisdom www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 2

Garfield & Ben Zvi (2008) Big Ideas of Statistics Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 3 Garfield & Ben Zvi (2008) Big Ideas of Statistics Reasoning about Data Reasoning about Models & Modeling Reasoning about Distribution Reasoning about Center Reasoning about Variability Reasoning about Comparing Groups Reasoning about Samples & Sampling Reasoning about Statistical Inference Reasoning about Covariation http://download.springer.com/static/pdf/214/bfm%253A978-1-4020-8383-9%252F2%252F1.pdf www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 3

Ambiguity of Importance Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 4 Ambiguity of Importance Important as: a topic (randomness) or a claim: ME ~ 1/sqrt(n) A source for the ideas/relations in a discipline; a source of extensive social benefit or cost; or a source of cognitive misunderstanding (fallacy). In this talk, importance is a claim involving a fallacy, or extensive social benefit or cost (contribution or catastrophe) www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 4

Ambiguity of Importance Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 5 Ambiguity of Importance Topic (randomness) or a claim: ME ~ 1/sqrt(n) This paper focuses on claims or relationships having substantial social or cognitive consequences. www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 5

The Most Dangerous Equation Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 6 The Most Dangerous Equation In Picturing the Uncertain World, Howard Wainer argued that de Movire’s equation was the most dangerous equation in the world – among those that are unknown or ignored. Wainer gave six great examples. http://www.statlit.org/Wainer.htm www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 6

Utts (2003) 7 Things Citizens Should Know Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 7 Utts (2003) 7 Things Citizens Should Know Association vs. causation: Clinical trial (random assign) vs. observational study (confounding) Statistical significance vs practical importance ‘No effect’ vs ‘no significant effect’ Types/sources of bias Coincidences can be common Confusion of the inverse Normal vs. average https://www.ics.uci.edu/~jutts/AmerStat2003.pdf www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 7

#1: Statistics are Numbers in Context Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 8 #1: Statistics are Numbers in Context “Statistics are just numbers” fallacy. Numbers are facts – and so are statistics. Isaacson (2012): Where do Statistics come from. www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 8

#1A: Statistical Fallacies Probability Fallacies Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 9 #1A: Statistical Fallacies Probability Fallacies Confusion of the inverse: P(A|B) = P(B|A) C.f., Medical Tests: Chance that a diseased person will test positive vs. chance that a person testing positive has the disease. Conjunction fallacy: P(A&B) > P(A) Chance Linda is a bank teller and active feminist is greater than being a bank teller. P(A&B |C) > P(A |B&C): Three-factor fallacy www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 9

#1B: Statistical Fallacies Individuals vs. Groups Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 10 #1B: Statistical Fallacies Individuals vs. Groups Individual fallacy: From individuals to group The rich are more likely to vote Republican than the poor. Yet richer states tend to vote Democrat. Ecological fallacy: From group to individuals 3. Simpson’s Paradox. From groups to subgroups or vice-versa. www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 10

#1B2: Statistical Fallacies Ecological Fallacy Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 11 #1B2: Statistical Fallacies Ecological Fallacy . www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 11

#1B3: Statistical Fallacies Simpson’s Paradox (Before) Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 12 #1B3: Statistical Fallacies Simpson’s Paradox (Before) . www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 12

#1B3: Statistical Fallacies Simpson’s Paradox (After) Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 13 #1B3: Statistical Fallacies Simpson’s Paradox (After) . www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 13

#1D: Statistical Fallacies Coincidence-Causation Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 14 #1D: Statistical Fallacies Coincidence-Causation Any statistically-significant event/connection is evidence of causation. Coincidence is too unlikely to be just chance. Law of Very Large Numbers (Qual/Quant) * Unlikely is almost certain given enough tries * If P = 1/N, event is more likely than not in N tries www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 14

#1D: Statistical Fallacies Coincidence-Causation Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 15 #1D: Statistical Fallacies Coincidence-Causation Law of Truly Large Numbers is “sometimes called the Jeane Dixon effect (see also Postdiction)”. It holds that the more predictions a psychic makes, the better the odds that one of them will "hit". Thus, if one comes true, the psychic expects us to forget the vast majority that did not happen.” https://en.wikipedia.org/wiki/Law_of_truly_large_numbers www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 15

#1D: Statistical Fallacies Coincidence-Causation Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 16 #1D: Statistical Fallacies Coincidence-Causation A Swedish study in 1992 looked at the incidence of poor health (800 ailments) among those living close to high-voltage power lines over a 25-year period. The study found that the incidence of childhood leukemia was four times higher among those that lived closest to the power lines. They failed to compensate for the look-elsewhere effect; in any collection of 800 random samples, it is likely that at least one will be at least 3 standard deviations above the expected value, by chance alonehttps://en.wikipedia.org/wiki/Law_of_truly_large_numbers www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 16

#1A: Statistical Fallacies Non-Traditional: Confounding Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 17 #1A: Statistical Fallacies Non-Traditional: Confounding “Statistics are just numbers” Statistical significance is permanent! Permanent in repeated trials Permanent regardless of context Any/every observational association can be nullified/reversed by an unknown confounder www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 17

#2A: Statistical Principles Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 18 #2A: Statistical Principles De Movire’s equation SE: independent of size of population. SE ~ 1/Sqrt(n) Applications: Hot spots, coincidences, Birthday problem. www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 18

#2B: Statistical Principles Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 19 #2B: Statistical Principles Law of Very Large Numbers. Qualitative. The unlikely is almost certain given enough tries. Quantitative: An outcome is more likely than not given N tries when P = 1/N. Applications: Hot spots, coincidences, Birthday problem. www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 19

#2B: Algebra in 8th Grade is Better Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 20 #2B: Algebra in 8th Grade is Better Overall, college attendance was 35% more prevalent among Algebra 8 students (62%) than Math 8 (46%). P < .001 For students with similar math scores, college attendance was 32% more prevalent among Algeba 8 students (45%) than Math 8 (34%) but the difference in rates was not statistically significant. (samples of 128 vs. 136). http://files.eric.ed.gov/fulltext/EJ753970.pdf www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 20

#2B: Keyes “Seven Countries” Study Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 21 #2B: Keyes “Seven Countries” Study 1958: Countries with the highest fat consumption had the most heart disease . This study supported the “Fat is bad” health recommendations and the introduction of “low- fat” foods (which tended to be “high-carb” foods) https://authoritynutrition.com/modern-nutrition-policy-lies-bad-science/ www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 21

#2B: Plausibility versus provability Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 22 #2B: Plausibility versus provability Most journalistically significant findings are based on observational studies and involve associations that are plausible. But, 80-90% of the claims coming from supposedly scientific studies in major journals fail to replicate.   They can’t be scientifically proven. http://www.forbes.com/sites/henrymiller/2014/01/08/the-trouble-with-scientific- research-today-a-lot-thats-published-is-junk/#482fd39520b8 www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 22

#2B: Benefit of Observational Studies Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 23 #2B: Benefit of Observational Studies “Observational studies are only good for generating hypotheses.” https://feinmantheother.com/2012/07/11/reading-the-scientific-literature-a-guide-to- flawed-studies/ No! Good observational studies are needed where randomization and treatment is impossible, unethical or unfeasible. www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 23

Cornfield Conditions In replying to Fisher, Cornfield proved a necessary condition for a confounder to nullify (or reverse) an observed association. “Cornfield's minimum effect size is as important to observational studies as is the use of randomized assignment to experimental studies.” Schield (1999) Simpson’s Paradox & the Cornfield Conditions www.statlit.org/pdf/1999SchieldASA.pdf

Contribution: The Cornfield Conditions Data showed that smokers were 10 times as likely to develop lung cancer as were non-smokers. Some statisticians wanted to support the claim that smoking “caused” lung cancer. Sir Ronald Fisher (1958) noted that “association was not causation” and that there was a difference (factor of two) in smoking preference between fraternal and identical twins. Cornfield et al (1959) argued that to nullify or reverse the observed association, the relative risk of a confounder must exceed the relative risk of that association. Fisher never replied.

Stratification Two-Way Half Tables Analyzing Numbers in the News StatLit for Managers Statistical Literacy for ManagersStatLit for Managers 15 May 2008 2013 1 March 20132013 Stratification Two-Way Half Tables Patient Died “Good” “Poor” TOTAL City Hospital 1% 6% 5.5% Rural Hospital 2% 7% 3.5% 1.5% 6.5% Patient at City is 2 pts more likely to die that at Rural. Patient in Poor condition is 5 pts more likely to die than is a Patient in Good condition. Association with Outcome: Confounder > Predictor 2008SchieldNNN6up.pdf 2013Schield-MBAA 26 26 26 www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA

Stratification Two-Way Half Tables Analyzing Numbers in the News StatLit for Managers Statistical Literacy for ManagersStatLit for Managers 15 May 2008 2013 1 March 20132013 Stratification Two-Way Half Tables Patient Died “Good” “Poor” TOTAL City Hospital 1% 6% 3% Rural Hospital 2% 7% 1.2% 3.8% 2.7% Patient at City is 2 pts more likely to die that at Rural. Patient in Poor condition is 5 pts more likely to die than is a Patient in Good condition. Association with Outcome: Confounder > Predictor 2008SchieldNNN6up.pdf 2013Schield-MBAA 27 27 27 www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA

Cornfield Condition for Nullification or Reversal Schield (1999) based on realistic data

Cornfield Condition for Nullification or Reversal Schield (2004) IASE

Cornfield Condition for Nullification or Reversal An association is nullified or reversed only if confounder (patient condition) has a stronger association with the outcome (death) than does the predictor (hospital). predictor (hospital) has a stronger association with the confounder (patient condition) than with the outcome (death).

Effect Sizes: Relative Risk Obese vs. non-Obese Chambers and Wakley (2002). Obesity and Overweight Matters in Primary Care

Confounder Distribution: Simple One-Parameter Model Assume: RR of confounders is distributed exponentially with a minimum RR of one and a mean RR of two.

Effect Sizes: Relative Risk 95% Confounder Resistant Obese vs. non-Obese

Contributions of Statistics to Human Knowledge Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 34 Contributions of Statistics to Human Knowledge . www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 34

Statistical Literacy for ManagersStatLit for Managers 2013 1 March 20132013 35 #2B: More Math is Better Math is a gatekeeper. The highest math class a high school senior takes has a major influence on both college acceptance and college choice. ” President, Calif School Boards Association, 2014 www.csba.org/Newsroom/CSBANewsletters/2014/May/InPrint/2014_MayCSN_VantagePt.aspx students who completed algebra in the eighth grade stayed in the mathematics pipeline longer and attended college at greater rates than those who did not. www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 35

#1C: Statistical Fallacies Lieberson (1985) Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 36 #1C: Statistical Fallacies Lieberson (1985) the selectivity problem due to pseudo-controls, contamination of control group by treatment, asymmetric causation (irreversible processes), Using high R2 as goal of a good explanation Presuming that adding more control variables takes one closer to the truth. Lieberson, S. (1985). Making It Count: The Improvement of Social Research and Theory. University of California Press. www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 36

#2B: Smaller Class Sizes are Better Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 37 #2B: Smaller Class Sizes are Better … www.centerforpubliceducation.org/Main-Menu/Organizing-a-school/Class-size-and-student-achievement-At-a-glance/Class-size-and-student-achievement-Research-review.html www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 37

Augsburg Student Survey: Seven Most Important Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 38 Augsburg Student Survey: Seven Most Important 1 All sources of influence (Take CARE) 2 Confounding 2 Hypothetical thinking: confounders, definitions. 4 Statistics are more than numbers. 5 Association-causation & Randomness (Luck vs. skill) 5 Bias: Placebo, Single blind; double blind 5 Named Ratios grammar; Percent, Percentages, Rates www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 38

Statistical Literacy for ManagersStatLit for Managers 2013 1 March 20132013 39 Conclusion Introductory statistics must be re-engineered: Allow for differences in students aptitudes Allow for difference in student interests Increase focus on multivariate & confounding Increase focus on Context: Where do Stats come from? www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 39

Statistical Literacy for ManagersStatLit for Managers 2013 1 March 20132013 40 References McKenzie, John, Jr. (2004) . Teaching the Core Concepts. ASA www.statlit.org/pdf/2004McKenzieASA.pdf Schield, M. (2015). Statistical Inference for Managers. ASA www.statlit.org/pdf/2015-Schield-ASA.pdf Schield, M. (2014). Two Big Ideas for Teaching Big Data: ECOTS. www.statlit.org/pdf/2014-Schield-ECOTS.pdf Berendsen, Hadlich and van Amersfoort (2011). Is Conjunction Fallacy Really a Fallacy? http://bacon.umcs.lublin.pl/~lukasik/wp- content/uploads/2010/12/Looking-at-Linda.pdf www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 40

McKenzie (2004) Core Concepts in Intro Stats Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 41 McKenzie (2004) Core Concepts in Intro Stats McKenzie (2004) asked statistical educators to pick the top-three core concepts in intro statistics: 75% Variation 31% Association vs. causation 25% Hypothesis tests 24% Sampling distribution 22% Confidence intervals 14% Randomness and statistical significance %: Percentage of votes by Statistical Educators Sample size: 56 95% ME = 12 percentage points www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 41

Verkuilen@UIUC (2013) Big Ideas in Probability/Stats Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 42 Verkuilen@UIUC (2013) Big Ideas in Probability/Stats Law of Large Numbers Central Limit Theorem Additional Big Ideas: de Moivre's Equation The Gauss–Markov theorem Cochran's theorem. https://www.quora.com/What-are-the-Big-Ideas-in-probability-statistics www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 42

Chen@Harvard (2014) Big Ideas in Stat 111 Theory Statistical Literacy for ManagersStatLit for Managers StatLit for Managers 2013 1 March 20132013 43 Chen@Harvard (2014) Big Ideas in Stat 111 Theory Bayes rule and Data generation Likelihood functions Point estimators: MLE, MOME, MAP Interval estimates: Exact, Asymptotic, etc. Calculus: Transformation, Lagrange, MLE Sufficient statistics; pivotal quanitites Bias, Variance, Information Asymptotic behavior: MLE, Bayes posterior Power & Hyp. Testing. Sample size www.quora.com/What-are-the-top-10-big-ideas-in-Statistics-111-Introduction-to-Theoretical-Statistics-at-Harvard www.StatLit.org/pdf/2013-Schield-MBAA-6up.pdf2013Schield-MBAA 2013Schield-MBAA 43