1. Chapter 9 Comparing More than Two Means

2. Review of Simulation-Based Tests  One proportion:  We created a null distribution by flipping a coin, rolling a die, or some computer simulation.  We then found where our sample proportion was in this null distribution.

3. Simulation-Based Tests  Comparing two proportions: Assuming there was no was no association between explanatory and response variables (the difference in proportions is zero), we shuffled cards and dealt them into two piles. (This essentially scrambled the response variable.) We then calculated the difference in proportions many times and built a null distribution. We finally found where the difference in our original sample proportions was located in the null distribution.

4. Simulation-Based Tests  Comparing two means: Assuming there was no relationship between explanatory and response variables (so the difference in means should be zero), we scrambled the response variable and calculated the difference in means many times and built a null distribution. We the found where the difference in our original two sample means was located in the null distribution.

5. Simulation-Based Tests  Paired Test: Assuming there was no relationship between the explanatory and response variables (so the mean difference should be zero), we randomly switched some of the pairs and calculated the mean of the differences many times and built a null distribution. We then found where the original mean of the differences from the sample was located in the null distribution.

6. Simulation-Based Tests  Comparing more than two proportions: Assuming there was no was no association between explanatory and response variables (all the proportions are the same), we scrambled the response variable and calculated the MAD statistic (or χ 2 statistic) many times and built a null distribution. We finally found where the original MAD or χ 2 statistic from our sample was located in the null distribution.

7. Two more types of tests  We now want to compare multiple means (more than two).  In chapter 10 we will look at an association between two quantitative variables using correlation and regression.  Both of these processes are basically the same as most of the simulation-based tests we have already done. Just the data types and the statistic we use is different.

8. Follow up tests  In the last chapter, we tested multiple proportions and if we found significance, we followed this up with by calculating confidence intervals to find out exactly which proportions were different.  Why didn’t we just start out finding a number of confidence intervals?  Let’s go through the following example to answer this and introduce tests for multiple means.

9. Section 9.1. Comparing Multiple Means: Simulation-Based Approach  Suppose we wanted to compare how much various energy drinks increased people’s pulses.  We would end up with a number of means. (Caffiene amounts shown are mg per 12 oz.) 55 120 250

10. Controlling for Type I Error  We could do this with multiple tests where we compared two means at a time, but If we were comparing 3 means, we would have to use 3 two-sample tests to compare these three means. (A vs B, B vs C, and A vs C)  If each test has a 5% significance level, there’s a 5% chance making a Type I Error. (We can call this a false alarm. Rejecting the null when it is true. There really is no difference between our groups and we got a result out in the tail just by chance alone.)

11. Controlling for Type I Error  These type I errors “accumulate” when we do more tests on the same data. At the 5% significance level, the probability of making at least one type I error for three test would be 14%. Comparing 4 means (6 tests), this jumps to 26%. Comparing 5 means (10 tests), this jumps to 40%.  An alternative approach uses one over-all test that compares all means at once.

12. Overall Test  We used one overall test in the last chapter when we compared proportions and we will do the same for comparing means.  If I have two means to compare, we just need to look at their difference to measure how far apart they are.  Suppose we wanted to compare three means. How could I create something that would measure how different all three means are?

13. A measure to compare 3 means  We will use the same MAD statistic as before, but this time look at the mean absolute differences for averages.  MAD = (|avg1 – avg2|+|avg2 – avg3|+ |avg3 – avg1|)/3  Let’s try this on an example!

14. Comprehension Example  Students were read an ambiguous prose passage under one of the following conditions: Students were given a picture that could help them interpret the passage before they heard it. Students were given the picture after they heard the passage. Students were not shown any picture before or after hearing the passage.  They were then tested on their comprehension of the passage.

15. Comprehension Example  This experiment is a partial replication done here at Hope of a study done by Bransford and Johnson (1972).  The students were randomly assigned to one of the three groups.  They listened to the passage with either a picture before, a picture after, or neither.

16. Hypotheses  Null: In the population there is no association between whether or when a picture was shown and comprehension of the passage  Alternative: In the population there is an association between whether and when a picture was shown and comprehension of the passage

17. Hypotheses  Null: All three of the long term mean comprehension scores are the same. µ no picture = µ picture before = µ picture after  Alternative: At least one of the mean comprehension scores is different.

18. Results Means 3.37 3.21 4.95

19. Finding the measure MAD = (|3.21−4.95|+|3.21−3.37|+|4.95−3.37|)/3 = (1.74 + 0.16 + 1.58)/3 = 3.48/3 = 1.16.  What is the likelihood of this happening by chance if there were really no difference in comprehension between the three groups?  What types of values (e.g., large, small, positive, negative) of this statistic will give evidence against the null hypothesis?

20. Let’s test this  Get the data from the website.  Go to the Analyzing Quantitative Response Applet and paste in the data.  Run the test. This applet is very similar to the one we used in the previous chapter.

21. Conclusion  Since we have a small p-value we can conclude at least one of the mean comprehension scores is different.  Can we tell which one or ones?  Go back to dotplots and take a look.  We can do pairwise confidence intervals to find which means are significantly different than the other means and will do that in the next section.

22. Expl 9.1: Exercise and Brain Volume  Brain size usually shrinks as we age and such shrinkage may be linked to dementia.  Can we do something to protect against this shrinkage?  A study done in China randomly assigned elderly volunteers to one four groups: tai chi, walking, social interaction, none.  Percentage of brain size increase or decrease was calculated after the study.

23. Section 9.2 Theory-Based Approach to Compare Multiple Means (ANalysis Of Variance ANOVA)

24. ANOVA  Like in chapter 8 when we compared multiple proportions, we need a statistic other than the MAD to make the transition to theory- based a smooth one.  This new statistic is called an F statistic and the theory-based distribution that estimates our null distribution is called an F distribution.  Unlike the MAD statistic, the F statistic takes into account the variability within each group.

25. F test statistic  The analysis of variance F test statistic is:

26. F test statistic  Remember measures of variation are always non- negative. (Our measure of variation can be zero when all values in the data set are the same.)  So our F statistic is also non-negative

27. F test statistic  Remember our ambiguous prose example? The researchers also had the students take a recall test several hours later to measure how well they could recall the content of the passage.

28. The difference in means matters and so does the individual group’s variation.  Original recall data on the left, hypothetical recall data on the right. Variation between groups is the same, variation within groups is different. How will this affect the F test statistic?

29. Hypotheses  Null: All three of the long-run mean recall scores for students under the different conditions are the same. (No association)  Alternative: At least one of the long-run mean recall scores for students under the different conditions is different. (Association)

30. Theory-Based ANOVA test  Just as with the simulation-based method, we are assuming we have independent groups.  Two extra conditions must be met to use traditional ANOVA: Normality: If sample sizes are small within each group, data shouldn’t be very skewed. If it is, use simulation approach Equal variation: Standard deviations of each group should be within a factor of 2 of each other

31. F test statistic  Are these conditions met for our recall data?

32. Let’s run the test  Let’s get the Recall data and run the test using the same applet we used last time (Analyzing Quantitative Response)  Let’s do simulation using the MAD statistic as well as the F statistic.  Then do theory-based methods using ANOVA.  If we get a small p-value, we will follow this overall test up with confidence intervals to determine exactly where the difference occurs.

33. Conclusion  Since we have a small p-value we have strong evidence against the null and can conclude at least one of the long-run mean recall scores is different.  From our confidence intervals,  After - Before: (-4.05, -1.74)*  After - None: (-2.42, -0.11)*  Before - None: (0.4756, 2.7875)*  We can see that each is significant so µ picture after ≠ µ picture before µ pictureafter ≠ µ no picture µ picture before ≠ µ no picture

34. Strength of Evidence  As sample size increases, strength of evidence increases.  As the means move farther apart, strength of evidence increases. (This is the variability between groups.)  As the standard deviations increase, strength of evidence decreases. (This is the variability within groups.)

35. Exploration Exploration 9.2: Comparing Popular Diets