2. ANOVA 3/19/12 Mini Review of simulation versus formulas and theoretical distributions Analysis of Variance (ANOVA) to compare means: testing for a difference in means between multiple groups Section 8.1 Professor Kari Lock Morgan Duke University

3. Anonymous Midterm Evaluation (due TODAY, 5pm) Project 1 (due Thursday, 3/22, 5pm) Project 1 Homework 7 (due Monday, 3/26) Homework 7 NO LATE HOMEWORK ACCEPTED! Turn in by Friday, 3/23, 5pm to get it graded before Exam 2. To Do

4. Two Options for p-values We have learned two ways of calculating p-values: The only difference is how to create a distribution of the statistic, assuming the null is true: 1)Simulation (Randomization Test): Directly simulate what would happen, just by random chance, if the null were true 2)Formulas and Theoretical Distributions: Use a formula to create a test statistic for which we know the theoretical distribution when the null is true, if sample sizes are large enough

5. Two Options for Intervals We have learned two ways of calculating intervals: 1)Simulation (Bootstrap): Assess the variability in the statistic by creating many bootstrap statistics 2)Formulas and Theoretical Distributions: Use a formula to calculate the standard error of the statistic, and use the normal or t- distribution to find z* or t*, if sample sizes are large enough

6. Pros and Cons 1)Simulation Methods PROS: Methods tied directly to concepts, emphasizing conceptual understanding Same procedure for every statistic No formulas or theoretical distributions to learn and distinguish between Works for any sample size Minimal math needed CONS: Need entire dataset (if quantitative variables) Need a computer Newer approach, so different from the way most people do statistics

7. Pros and Cons 2)Formulas and Theoretical Distributions PROS: Only need summary statistics Only need a calculator The approach most people take CONS: Plugging numbers into formulas does little for conceptual understanding Many different formulas and distributions to learn and distinguish between Harder to see the big picture when the details are different for each statistic Doesn’t work for small sample sizes Requires more math and background knowledge

8. Two Options If the sample size is small, you have to use simulation methods If the sample size is large, you can use whichever method you prefer It is redundant to use both methods, unless you want to check your answers

9. Accuracy The accuracy of simulation methods depends on the number of simulations (more simulations = more accurate) The accuracy of formulas and theoretical distributions depends on the sample size (larger sample size = more accurate) If the sample size is large and you have generated many simulations, the two methods should give essentially the same answer

10. Multiple Categories So far, we’ve learned how to do inference for a difference in means IF the categorical variable has only two categories Today, we’ll learn how to do hypothesis tests for a difference in means across multiple categories

11. Hypothesis Testing 1.State Hypotheses 2.Calculate a statistic, based on your sample data 3.Create a distribution of this statistic, as it would be observed if the null hypothesis were true 4.Measure how extreme your test statistic from (2) is, as compared to the distribution generated in (3) test statistic

12. Hypotheses To test for a difference in means across k groups:

13. Test Statistic Why can’t use the familiar formula to get the test statistic? More than one sample statistic More than one null value We need something a bit more complicated…

14. Difference in Means Whether or not two means are significantly different depends on How far apart the means are How much variability there is within each group

15. Difference in Means

16. Analysis of Variance Analysis of Variance (ANOVA) compares the variability between groups to the variability within groups Total Variability Variability Between Groups Variability Within Groups

17. Analysis of Variance If the groups are actually different, then a)the variability between groups should be higher than the variability within groups b)the variability within groups should be higher than the variability between groups

18. Discoveries for Today How to measure variability between groups? How to measure variability within groups? How to compare the two measures? How to determine significance?

19. Notation k = number of groups n j = number of units in group j n = overall number of units = n 1 + n 2 + … + n k

20. Discoveries for Today How to measure variability between groups? How to measure variability within groups? How to compare the two measures? How to determine significance?

21. Sums of Squares We will measure variability as sums of squared deviations (aka sums of squares) familiar?

22. Sums of Squares Total Variability Variability Between Groups Variability Within Groups overall mean data value i overall mean mean in group j i th data value in group j Sum over all data valuesSum over all groups Sum over all data values

23. Deviations Group 1 Group 2 Overall Mean Group 1 Mean

24. Sums of Squares Total Variability Variability Between Groups Variability Within Groups SST (Total sum of squares) SSG (sum of squares due to groups) SSE (“Error” sum of squares)

25. Source Groups Error Total df k-1 n-kn-k n-1 Sum of Squares SSG SSE SST Mean Square MSG = SSG/(k-1) MSE = SSE/(n-k) ANOVA Table The “mean square” is the sum of squares divided by the degrees of freedom variability average variability

26. Discoveries for Today How to measure variability between groups? How to measure variability within groups? How to compare the two measures? How to determine significance?

27. F-Statistic The F-statistic is a ratio of the average variability between groups to the average variability within groups The F-statistic is the test statistic for testing for a difference in means across more than 2 groups

28. Source Groups Error Total df k-1 n-kn-k n-1 Sum of Squares SSG SSE SST Mean Square MSG = SSG/(k-1) MSE = SSE/(n-k) F Statistic MSG MSE ANOVA Table

29. Jumping and Bone Density Does jumping improve bone density? 30 rats were randomized to three treatment groups: No jumping (10 rats - group 1) 30 cm jump (10 rats - group 2) 60 cm jump (10 rats - group 3) Rats performed 10 jumps per day, 5 days per week. Bone density was measured after 8 weeks.

30. Jumping and Bone Density mean sd n Control 601.1 27.36360 10 Lowjump 612.5 19.32902 10 Highjump 638.7 16.59351 10

31. Jumping and Bone Density

32. Source Groups Error Total df k-1 = 2 n-k =27 n-1=29 Sum of Squares 7433.9 12579.5 20013.4 Mean Square 7433.9/2 =3716.9 12579.5/27 = 465.9 F Statistic 3716.9 465.9 =7.98 ANOVA Table

33. Discoveries for Today How to measure variability between groups? How to measure variability within groups? How to compare the two measures? How to determine significance?

34. We have a test statistic. What else do we need to perform the hypothesis test? A distribution of the test statistic assuming H 0 is true How do we get this? Two options: 1)Simulation 2)Distributional Theory

35. F-statistic If there really is a difference between the groups, we would expect the F-statistic to be a)Higher than we would observe by random chance b)Lower than we would observe by random chance If the null hypothesis is true, what kind of F- statistics would we observe just by random chance?

36. Simulation Rerandomize (reallocate) the rats to treatment groups, keeping response values fixed Calculate the F-statistic Repeat this simulation many times to form a randomization distribution Calculate the p-value as the proportion as extreme or more extreme than the observed F-statistic

37. F-distribution p-value = 0.002 Because a difference in groups would make the F-statistic higher, calculate probability in the upper tail

38. F-distribution

39. F-Distribution If the following conditions hold, 1.Sample sizes in each group are large (each n j ≥ 30) OR the data are relatively normally distributed 2.Variability is similar in all groups 3.The null hypothesis is true then the F-statistic follows an F-distribution The F-distribution has two degrees of freedom, one for the numerator of the ratio (k – 1) and one for the denominator (n – k) http://www.capdm.com/demos/software/html/capdm/qm/fdist/usage.html

40. Equal Variance The F-distribution assumes equal within group variability for each group As a rough rule of thumb, this assumption is violated if the standard deviation of one group is more than double the standard deviation of another group

41. F-distribution Can we use the F-distribution to calculate the p-value for the jumping and bone density F-statistic? a)Yes b)No c)I need more information mean sd n Control 601.1 27.36360 10 Lowjump 612.5 19.32902 10 Highjump 638.7 16.59351 10

42. Jumping and Bone Density

43. F-distribution p-values 1)Online applet: http://www.danielsoper.com/statcalc3/calc.aspx?id=7 http://www.danielsoper.com/statcalc3/calc.aspx?id=7 2)RStudio: > pf(7.98,2,27,lower.tail=FALSE) [1] 0.001892532 3) TI-83: 2 nd  DISTR  9:Fcdf(  7.98, 9999, 2, 27

44. Source Groups Error Total df k-1 n-kn-k n-1 Sum of Squares SSG SSE SST Mean Square MSG = SSG/(k-1) MSE = SSE/(n-k) F Statistic MSG MSE p-value Use F k-1,n-k ANOVA Table

45. Source Groups Error Total df 2 27 29 Sum of Squares 7433.9 12579.5 20013.4 Mean Square 3716.9 465.9 F- Statistic 7.98 ANOVA Table p-value 0.0019 We have strong evidence that jumping does increase bone density, at least in rats.

46. Study Hours by Class Year Is there a difference in the average hours spent studying per week by class year at Duke? (a)Yes (b)No (c)Cannot tell from this data (d)I didn’t finish

47. Source Groups Error Total df 2 195 197 Sum of Squares 318 24984 20013.4 Mean Square 159 128.1 F- Statistic 1.24 ANOVA Table p-value 0.29

48. Summary Analysis of variance is used to test for a difference in means between groups by comparing the variability between groups to the variability within groups Sums of squares are used to measure variability The F-statistic is the ratio of average variability between groups to average variability within groups The F-statistic follows an F-distribution, if sample sizes are large (or data is normal), variability is equal across groups, and the null hypothesis is true