Step 1: Specify a null hypothesis We have 8 books and we wish to see whether the books differ on the basis of their average sentence count. If i represents the mean of the ith book, then our null hypothesis is Statistical Data Analysis - Lecture11 28/03/03

Step 2: Specify an alternative hypothesis Statistical Data Analysis - Lecture11 28/03/03 Step 2: Specify an alternative hypothesis The null hypothesis is a fairly general one. The reason for this is to avoid multiple comparison problems. To reject the null hypothesis, only one mean has to differ, i.e. Statistical Data Analysis - Lecture11 28/03/03

Statistical Data Analysis - Lecture11 28/03/03 Model fit diagnostics We did some “pre-fit” checking of the assumptions of ANOVA However, we need to do some further checks after fitting our model Both of these posthoc methods involve examination of the residuals Recall the residuals are the small “errors” left after we subtract the estimated group means from each observation The first plot is called a “pred-res” plot, or a residuals vs.. predicted plot. Statistical Data Analysis - Lecture11 28/03/03

Statistical Data Analysis - Lecture11 28/03/03 Pred-res plots Pred-res plots are highly patterned in a sense for ANOVA. The reason is that the fitted value is the same for each observation in a group, i.e. it is the group mean So, we expect to get a plot with the group means on the x-axis Recall that we assumed that all the residuals have mean zero and standard devotion . Furthermore, we assumed that the residuals are normally distributed Therefore we expect to the points for each group to be roughly centred around zero (symmetric) spread out about the same amount not to have too many extreme points Statistical Data Analysis - Lecture11 28/03/03

Statistical Data Analysis - Lecture11 28/03/03 1.8 2.0 2.2 2.4 2.6 -3 -2 -1 1 2 Fitted values Residuals Residuals vs. Fitted 300 41 286 aov(formula = log.length ~ book) Statistical Data Analysis - Lecture11 28/03/03

Interpreting pred-res plots We can see that things are basically okay with our assumptions The data are symmetric around zero for the most part There don’t appear to be any extreme values The WGMS is 0.507. Remember this is an estimate of the variance of the residuals, so the standard deviation is the square root of the WGMS. This is 0.712, so we expect 99.9% of the residuals to be with in 3 standard deviations, or within +/- 2.136 There is one observation that exceeds this bound. If we divide by the square root of the WGMS, we get an approximate Z-score of about –3.66. This says that this point is 3.66 std. deviations from the mean, which is extreme, but not too bad. Statistical Data Analysis - Lecture11 28/03/03

Statistical Data Analysis - Lecture11 28/03/03

Statistical Data Analysis - Lecture11 28/03/03 Norplots of residuals Here is a situation where we need the use of a norplot We use norplot to test the assumption that the residuals are normally distributed From the pred-res plot we can determine things like symmetricity of the data and inequality of variance The calculations on the previous slide are interesting, but in fact similar results are guaranteed by Komologorov’s theorem A norplot plots the the theoretical quantiles from the normal distribution vs. quantiles for the residuals (which are really just the order statistics). A straight line => Normality Statistical Data Analysis - Lecture11 28/03/03

Statistical Data Analysis - Lecture11 28/03/03

Interpreting the norplot From the norplot, even though the aspect ratio is not 1:1 we can see that the data pretty much follow a straight line, therefore our assumption of normality is sound’ There is one outlying point, and it may be interesting to investigate the effect of removing it from the fit Identifying points in R Easiest way is to make a vector of labels and then use the text function to add the labels to the plot Statistical Data Analysis - Lecture11 28/03/03

Interpreting the output We saw a P-value in our ANOVA table of 1.055E-14 this is a very strong implication that there is a difference between the authors. How do we find the differences? They’re just the group means which we can get using the apply (or sapply) function Whilst we might use this as a rough guide to identifying the differences, we really run into a multiple comparison problem Statistical Data Analysis - Lecture11 28/03/03

Multiple comparisons problem Every time we perform a hypothesis test, we set ourselves an artificial critical point or significance value, like 95% or 99%. In setting such a significance value we accept a certain Type I error rate or size. For example, if we use 95% we accept that we will reject the null hypothesis when it is actually true on average about 5% of the time. In a one way ANOVA we initially carry out one hypothesis test. So we have (assuming the use of a 95% significance level) a 5% chance of making the wrong conclusion if the null hypothesis is true An acceptable level? Statistical Data Analysis - Lecture11 28/03/03

Multiple comparisons problem If we stopped at that point, then yes, the Type I error rate would be fine. However, it is usually not sufficient to know that there is a difference, we want to know where the differences are We could compare pairs of groups at at time and look for differences that way. Is this a good thing to do? Yes and no. Every time we compare two groups, we have the same 5% chance of making an incorrect conclusion Because these tests are essentially independent, the number of Type I errors is binomially distributed Statistical Data Analysis - Lecture11 28/03/03

Type I error and Bonferroni corrections E.g for 5 groups, there are 5*4/2=10 possible comparisons, so the probability of making at least one mistake has risen to 40%! In general if there are k pairwise tests and the Type I error is  then the probability of making at least one mistake is In general if there are k pairwise tests and the Type I error is  then the probability of making at exactly one mistake is This can be approximated by k× Statistical Data Analysis - Lecture11 28/03/03

Type I error and Bonferroni corrections What can we do about it? One solution is called the Bonferroni correction. It involves dividing the Type I error rate by k. E.g. if we want an overall Type I error rate of 5% and we have 10 tests to carry out we set  = 0.05/10 = 0.005 This test is very conservative A conservative test means that it is very difficult to reject the null hypothesis. Statistical Data Analysis - Lecture11 28/03/03

Conservative tests and LSD intervals One analogy is that if your pants would have to be on fire before you noticed your house was burning down Bonferroni, especially for large numbers of groups, is far too conservative to be acceptable. What other procedures are there? One other is known as Fisher’s Protected Least Significant Difference or LSD intervals Statistical Data Analysis - Lecture11 28/03/03

Fisher’s protected LSD Fisher’s method is as follows If you have rejected the ANOVA null hypothesis of no difference between the means, then calculate confidence intervals for the pairwise differences between the means If two intervals overlap, then there is no evidence to suggest that there is a difference between the two groups (or treatments) If two intervals do not overlap, then there is some evidence at the  level of significance. How do we construct the intervals Statistical Data Analysis - Lecture11 28/03/03

Statistical Data Analysis - Lecture11 28/03/03 LSD intervals/plots For each of our k groups we calculate   where We usually plot these intervals for each group and make our interpretations from there This is a conservative approach and works well when the groups have similar numbers of observations. If the groups have less than 10 observations in each group and the numbers in each group differ greatly then this rule should be used with caution. Statistical Data Analysis - Lecture11 28/03/03

Interpreting LSD plots For every group we have an LSD interval. We compare one of these intervals with an LSD interval for another group. This is an approximation of performing a two sample t-test for a difference in the group means. There are two things that can happen: (a) the intervals overlap and (b) the intervals don’t overlap. When the intervals overlap (and given certain assumptions which we will discuss later) we can say at the 5% level of significance that the data provide no evidence against the null hypothesis, i.e. the data provide no evidence of a difference in the group means. Statistical Data Analysis - Lecture11 28/03/03

Non-overlapping intervals When the intervals don’t overlap (and given the same conditions), we can say at the 5% level of significance that the data provide sufficient evidence against the null hypothesis, i.e. the data provide evidence to suggest a difference in the means of the two groups. Statistical Data Analysis - Lecture11 28/03/03

Assumptions in LSD interval comparisons  When we use an LSD plot to make a pairwise comparison we assume two things   The numbers of observations per group are fairly similar – i.e. balanced or “near balanced” designs are better The standard errors of the groups are approximately equal. If there is any doubt about either of these assumptions, then any conclusions about differences or lack thereof should be backed up with a proper two sample t-test. Statistical Data Analysis - Lecture11 28/03/03