1. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly Significance and Sample Size Refresher Harrison W. Kelly III, Ph.D. Lecture # 3

2. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 2 Significance Statistical Significance An observed difference that is unlikely to have occurred by chance Practical Significance - The “ Real ” Question The point at which you decide that the difference that exists between two things is important enough to take action on

3. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 3 Hypothesis Testing The process of taking a practical problem translating it to a statistical problem Because we are using samples (and relatively small ones at that) to estimate population parameters, there is always a chance that we can select a “ weird ” sample for our experiment that may not represent a “ typical ” set of observations Because of this, inferential statistics with some assumptions allows us to estimate the probability of getting a “ weird ” sample

4. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 4 Hypothesis Testing Testing is performed on the Null Hypothesis, if the null hypothesis is not supported, we must default to the alternative hypothesis! We never actually test the hypothesis that we are biased in favor of, rather we show that the converse is not true, so therefore, our hypothesis should be supported The Null hypothesis (Ho) is assumed to be true, the data must provide sufficient evidence to support that the Null hypothesis is probably not true

5. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 5 Hypothesis Testing After data is collected, a test statistic is calculated A p-value is then calculated based on the test statistic The p-value tells you the the chance that the results would have occurred if Ho were true The probability that an event like the one observed would happen when sampling from a single population Correct Decision Type 2 Error (  Type 1 Error (  Ho Ho Ha HaTruthConclusion

6. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 6 Academic Example Say you study conditions conducive to potato rot by injecting potatoes with bacteria that cause rotting and subjecting them to two temperatures. You measure the percent of potatoes that have not rotted. We want to compare the average percent of unrotted potatoes at the two levels studied and determine if is different than the historical percentage of 13%. EXH_AOV.MTW

7. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 7 Academic Example Data RotTemp 1310 1110 31010 410 710 1510 210 710 2616 1916 2416 1516 2216 1816 2016 2416 816

8. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 8 Are the data Normally distributed?

9. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 9 Minitab Session Window Test for Equal Variances: Rot versus Temp 95% Bonferroni confidence intervals for standard deviations Temp N Lower StDev Upper 10 9 2.91929 4.55522 9.7110 16 9 3.54091 5.52519 11.7789 F-Test (normal distribution) Test statistic = 0.68, p-value = 0.598 Levene's Test (any continuous distribution) Test statistic = 0.05, p-value = 0.824 Asymmetric 95% confidence intervals based on the Chi-square distribution Both p-values are greater than reasonable choices of , so you fail to reject the null hypothesis

10. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 10 Minitab Output

11. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 11 Where are we? You fail to reject the null hypothesis of the variances being equal and conclude that there is no evidence for unequal variances. What do we need to know now?

12. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 12 Minitab Output Two-Sample T-Test and CI: Rot, Temp Two-sample T for Rot Temp N Mean StDev SE Mean 10 9 8.00 4.56 1.5 16 9 19.56 5.53 1.8 Difference = mu (10) - mu (16) Estimate for difference: -11.5556 95% CI for difference: (-16.6432, -6.4679) T-Test of difference = 0 (vs not =): T-Value = -4.84 P-Value = 0.000 DF = 15

13. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 13 Minitab Output

14. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 14 Minitab Output

15. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 15 Real-Life Example Say you had a gel dispense process and you were asked to find out with high confidence (95%) if the average dispense weight was similar to the target weight of 4 grams How would you you go about it?

16. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 16 How should we collect the data? Post-weigh parts or containers subtract the mean part or container weight Pre-weigh parts or containers, dispense, post-weigh parts or containers, calculate delta - bulk population Pre-weigh parts or containers, dispense, post-weigh parts or containers, calculate delta - individuals

17. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 17 Things to consider How will you physically capture the gel? What will you use as a balance? What kinds of data do you need? How much data do you need?

18. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 18 Key Phrases Say you had a gel dispense process and you were asked to find out with high confidence (95%) if the average dispense weight was similar to the target weight of 4 grams

19. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 19 95% Confidence Means that there is a 95% chance that your sample estimate will not be different from the population parameter Would you want to know the probability of detecting a difference from target if one is present? What is this known as? Does the question contain this information? Do we need this information to answer the question in an efficient way?

20. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 20 Power 1-  = the probability of detecting a difference if a difference is present to detect Let ’ s look at a range of powers (0.7, 0.8, and 0.9)

21. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 21 Similar to How similar to the target do we mean? We know that the average will not equal the target, so how different will we consider to be important enough so that we will take action Let ’ s use 10% of the target value or 0.4 grams

22. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 22 Summary so far We know: how confident we want to be in our estimate of mean weight how likely it is that we will detect a difference if a difference exists how big a difference we are concerned with What are we missing? Why do we need statistics? How much variability is there in the process?

23. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 23 Variability Affects the total number of samples we need to collect The greater the variability, the more samples needed to overcome the uncertainty caused by the variability Say for our example that our process variability is 1.0 gram

24. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 24 Minitab Output MTB > Power; SUBC> TOne; SUBC> Difference 0.4; SUBC> Power 0.7 0.8 0.9; SUBC> Sigma 1.0. Power and Sample Size 1-Sample t Test Testing mean = null (versus not = null) Calculating power for mean = null + difference Alpha = 0.05 Sigma = 1 Sample Target Actual Difference Size Power Power 0.4 41 0.7000 0.7052 0.4 52 0.8000 0.8078 0.4 68 0.9000 0.9016

25. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 25 Ok what do we do? We want to collect enough samples to answer the question We can ’ t (or don ’ t want to) afford the number of samples needed We can ’ t reduce the process variation arbitrarily We know that using 30 is not the right answer Let ’ s try one of the other Minitab options and see if they can help

26. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 26 Minitab Output Power and Sample Size 1-Sample t Test Testing mean = null (versus not = null) Calculating power for mean = null + difference Alpha = 0.05 Sigma = 1 Sample Difference Size Power 0.4 10 0.2051 0.4 15 0.3032 0.4 20 0.3970 0.4 25 0.4840 0.4 30 0.5628

27. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 27 Something has to give! We check with the engineer and find out that our real concern is detecting a change and knowing (>60% chance) that it has really changed Ideally, we would like to catch it as soon as it happens (small difference) but we could tolerate up to a 1 standard deviation difference Oh, we also find out that the engineer doesn ’ t want to spend lots of time or money on this project

28. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 28 Minitab Output Power and Sample Size 1-Sample t Test Testing mean = null (versus not = null) Calculating power for mean = null + difference Alpha = 0.05 Sigma = 1 Sample Difference Size Power 0.1 10 0.0593 0.2 10 0.0877 0.3 10 0.1361 0.4 10 0.2051 0.5 10 0.2932 0.6 10 0.3959 0.7 10 0.5064 0.8 10 0.6162 0.9 10 0.7172

29. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 29 Here ’ s what we decide Alpha = 0.05 Power = 62% Difference able to detect = 0.8 grams (about 20% of the target) Sample size = 10

30. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 30 Example So, if I go out and dispense 10 parts, I should be able to detect a difference from target of 0.8 grams with a 62% chance of detecting the change if the change exists! I generated 10 random samples using Minitab and the following parameters: MTB > random 10 c40; SUBC> normal 4.8 1. I should be able to detect this change from target

31. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 31 Minitab Output MTB > TTest 4 'SampRot'; SUBC> Alternative 0. T-Test of the Mean Test of mu = 4.000 vs mu not = 4.000 Variable N Mean StDev SE Mean T P SampRot 10 4.516 0.349 0.110 4.67 0.0012 It worked and I detected the change!

32. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 32 Had we collected 30 samples We could have: Detected a smaller difference Had greater confidence Had more power We could have: Collected to many samples (wasted time and money) Detected too small a difference

33. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 33 Consider Why we collected the data the way we did Why did we use measures of the same vessels before and after the dispense? What did we try to remove from the data set? This method of treating samples is related to another type of t test called a paired t test

34. Lecture Notes and Electronic Presentations, © 2013 Dr. Kelly 34 Sample Size In hypothesis testing, sample size is everything Too small a sample - risk not being able to determine a difference even if there is one Too large a sample - risk wasting too much money, time and resources showing that there is a difference