1. DATA CONFUSION How to confuse yourself and others with Data Analysis

2. AGENDA FOR TODAY’S TALK Good Graphs – Bad Graphs The Law of Averages PTBD Analysis Enumerative & Analytical Problems PARC Analysis Wrong Methods of Analysis

3. “There are three kinds of lies: Lies, damned lies and statistics” Attributed to Benjamin Disraeli by Mark Twain

4. GOOD GRAPHS AND BAD GRAPHS

5. DATA RELEVANCE Graphs are only as good as the data they display No amount of creativity can produce good graphs from dubious data

6. DATA CONTENT Don’t produce graphs from very small amounts of data The human brain can grasp 1, 2 or 3 numbers without a graph

7. RULES FOR PRODUCING GOOD GRAPHS KEEP IT SIMPLE AND STUPID –Jesse Ventura Tell the truth – don’t distort the data

8. GOOD GRAPHS Portray information without distortion Contain no distracting elements –No false third dimensions, irrelevant decoration, or colour (chartjunk) Use an appropriate scale Label axes and tick marks properly, including measurement units Have a descriptive title and/ or caption and legend Have a low ink – to – information ratio

9. BAD GRAPH GOOD GRAPH BAD GRAPH EVEN BETTER GRAPH

10. BAD GRAPH GOOD GRAPH

11. GRAPHS THAT CONFUSE

12. CHART JUNK

13. GRAPHS THAT TELL A STORY

14. HISTOGRAMS No meaningless gaps Reasonable Choice of bins Easy to choose or adjust bins Good aspect ratio Meaningful labels on axes Appropriate labels on bin tick marks

15. TRENDING RANDOM VARIATION “Upward trend” “Downturn” “Rebound” “Setback” “Turnaround” “Downward trend”

16. THE LAW OF AVERAGES “If I sit in a freezer and plunge my head into a pan of boiling chip fat..... on average, I’m quite comfortable.”

17. SHEWHART’S RULES FOR PRESENTATION OF DATA Rule One –Data should always be presented in a way that preserves the evidence in the data Rule Two –When an average, standard deviation or histogram is used to summarize data, the user should not be misled into to taking action they would not take if the data were presented in a time series

18. USING THE WRONG METHODS Descriptive Statistics: A, B, C, D Variable N Mean StDev CoefVar Minimum Maximum A 20 11.950 0.102 0.85 11.83 12.08 B 20 11.950 0.100 0.84 11.85 12.25 C 20 11.950 0.102 0.86 11.75 12.15 D 20 11.950 0.100 0.84 11.81 12.14 Process:ABCD 111.85 11.7512.14 211.8311.8611.9512.01 311.87 11.811.88 411.8411.8711.9412.07 511.8511.8811.95 611.8611.891211.87 711.8511.8912.0512.06 811.8511.911.8511.94 911.8411.9211.9411.84 1011.8611.9111.8512.05 1112.0511.9312.0511.93 1212.0611.9311.8511.83 1312.0311.9512.0512.04 1412.0211.9711.9511.92 1512.0311.9611.9511.82 1612.0411.9911.9512.03 1712.061211.8511.91 1812.061212.111.81 1912.0412.161212.01 2012.0812.2512.1511.81

19. NO SIGNIFICANT DIFFERENCE HERE!

20. NO DIFFERENCE?!?

21. ALWAYS CARRY OUT PTBD ANALYSIS PLOT THE B….. DOTS!

22. TYPES OF STATISTICAL STUDIES Descriptive Enumerative Analytic

23. DESCRIPTIVE STUDY Count all fish in barrel Count number of goldfish Proportion of goldfish applies to the fish population in this barrel and no other barrels of fish

24. ENUMERATIVE STUDY Take a sample of fish from the barrel, and count the number of goldfish in the sample Point estimate of the proportion of goldfish in the barrel population Many statistical procedures do this Cannot make any inference about any other barrels of fish

25. ANALYTICAL STUDY Will we get the same proportion of goldfish in the future as we got in the past? An analytical study allows prediction within limits Fish Packing Process over Time

26. ANALYTICAL STUDY Proportion of goldfish is stable over time Fish packing process is predictable within limits We can expect, on average, 4 goldfish per barrel, but as many as 10 and as few as 0 in any single barrel

27. ENUMERATIVE vs ANALYTICAL METHODS Enumerative methods –seek to provide numeric summaries, confidence intervals,etc –use significance tests, ANOVA, descriptive stats, etc., assume single, stable population Analytical methods –seek to understand the system under study –use primarily graphical tools such as run charts, control charts, histograms, box plots, etc –in the real world, most problems are analytical

28. “Analysis of variance, t-tests, confidence intervals, and other statistical techniques taught in books,….., are inappropriate because they provide no basis for prediction and because they bury the information contained in the order of production.” W.E. Deming, Out of the Crisis Traditional statistical methods have their place, but are widely abused in the real world. When this is the case, statistics do more to cloud the issue than to enlighten.

29. PARC ANALYSIS Practical Accumulated Records Compilation Passive Analysis (by) Regression Correlations Planning After Research Completed Profound Analysis Relying (on) Computers note inverse relationship with Continuous Recording (of) Administrative Procedures Constant Repetition (of) Anecdotal Perceptions

30. PLANNING A PROCESS IMPROVEMENT STUDY Why collect the data? What statistical methods for analysis? What data will be collected? How much data do we need? How will the data be measured? How good is the measurement system? When and where will data be collected? Who will collect the data? Remember:

31. GARBAGE IN – GARBAGE OUT

32. WHAT’S SIGNIFICANT? Two-sample T for C1 vs C2 N Mean StDev SE Mean A 5 13.652 0.487 0.22 B 5 14.369 0.646 0.29 Difference = mu (C1) - mu (C2) Estimate for difference: -0.716615 95% CI for difference: (-1.551531, 0.118301) T-Test of difference = 0 (vs not =): T-Value = -1.98 P-Value = 0.083 DF = 8 Both use Pooled StDev = 0.5725 Two-sample T for C3 vs C4 N Mean StDev SE Mean A 200 13.510 0.501 0.035 A 200 13.667 0.492 0.035 Difference = mu (C3) - mu (C4) Estimate for difference: -0.157292 95% CI for difference: (-0.254935, -0.059649) T-Test of difference = 0 (vs not =): T-Value = -3.17 P-Value = 0.002 DF = 398 Both use Pooled StDev = 0.4967 Mean A = 13.7, Mean B = 14.4 Not significant? Mean A = 13.5, Mean B = 13.7 Significant?

33. WHAT SHOULD I DO WITH OUTLIERS? Data point far away from the rest of the data Don’t remove outliers to make data “look good” Do you know why it is different? If you do, remove it. If you don’t, leave it in Could have a big impact on the analysis Re – run analysis without outlier, and compare results

34. “REGRESSION” WITH EXCEL Usually means drawing an X-Y plot, fitting a straight line and coming up with an R 2 value. As long as R 2 is high, everything’s hunky-dory. WRONG!

35. “REGRESSION” WITH EXCEL Relationship is clearly not linear, and should not be presented as such

36. “REGRESSION” WITH EXCEL Regression model checking – in Excel? Residual plots: –Normally distributed –Random pattern when plotted vs fitted values OKVariance not homogeneous Model incorrect

37. PITFALLS OF REGRESSION ANALYSIS Non-Linear Relationships Influential Points Extrapolating Lurking Variables Summary Data Assuming Causation

38. THAT’S (WITH REASONABLE PROBABILITY) THE END FOLKS! And remember, With statistics, you never have to say you’re certain!

39. THANK YOU FOR YOUR ATTENTION ARE THERE ANY QUESTIONS? GOOD LUCK!!