1. Treatment comparisons
ANOVA can determine if there are differences among the treatments, but what is the nature of those differences?
Are the treatments measured on a continuous scale?
Look at response surfaces (linear regression, polynomials)
Is there an underlying structure to the treatments?
Compare groups of treatments using orthogonal contrasts or a limited number of preplanned mean comparison tests
Are the treatments unstructured?
Use appropriate mean comparison tests

2. Comparison of Means Pairwise Comparisons
Least Significant Difference (LSD)
Simultaneous Confidence Intervals
Dunnett Test (making all comparisons to a control)
Bonferroni Inequality
Other Multiple Comparisons - “Data Snooping”
Fisher’s Protected LSD (FPLSD)
Student-Newman-Keuls test (SNK)
Tukey’s honestly significant difference (HSD)
Waller and Duncan’s Bayes LSD (BLSD)
False Discovery Rate Procedure
Often misused - intended to be used only for data from experiments with unstructured treatments

3. Multiple Comparison Tests
Fixed Range Tests – a constant value is used for all comparisons
Application
Hypothesis Tests
Confidence Intervals
Multiple Range Tests – values used for comparison vary across a range of means

4. Variety Trials
In a breeding program, you need to examine large numbers of selections and then narrow to the best
In the early stages, based on single plants or single rows of related plants. Seed and space are limited, so difficult to have replication
When numbers have been reduced and there is sufficient seed, you can conduct replicated yield trials and you want to be able to “pick the winner”

5. Least Significant Difference
Calculating a t for testing the difference between two means
any difference for which the t > t would be declared significant
Further, is the smallest difference for which significance would be declared
therefore
or with equal replication, where r is number of observations forming the mean
LSD t
MSE r = a 2 /

6. Do’s and Don’ts of using LSD
LSD is a valid test when
making comparisons planned in advance of seeing the data (this includes the comparison of each treatment with the control)
Comparing adjacent ranked means
The LSD should not (unless F for treatments is significant) be used for
making all possible pairwise comparisons
making more comparisons than df for treatments

7. Pick the Winner
A plant breeder wanted to measure resistance to stem rust for six wheat varieties
planted 5 seeds of each variety in each of four pots
placed the 24 pots randomly on a greenhouse bench
inoculated with stem rust
measured seed yield per pot at maturity

8. Ranked Mean Yields (g/pot)
Mean Yield Difference
Variety Rank Yi Yi-1 - Yi F D E B A C

9. ANOVA Compute LSD at 5% and 1% Source df MS F Variety 5 2,976.44 24.80
Error
Compute LSD at 5% and 1%
LSD t
MSE r = a 2
120 4
16.27 /
2.101 ( * )
2.878
22.29

10. Back to the data... Mean Yield Difference Variety Rank Yi Yi-1 - Yi
LSD=0.05 = 16.27
LSD=0.01 = 22.29
Mean Yield Difference
Variety Rank Yi Yi-1 - Yi F D
E * B
A *
C **

11. Pairwise Comparisons
If you have 10 varieties and want to look at all possible pairwise comparisons
that would be t(t-1)/2 or 10(9)/2 = 45
that’s a few more than t-1 df = 9
LSD would only allow 9 comparisons

12. Type I vs Type II Errors
Type I error - saying something is different when it is really the same (Paranoia)
the rate at which this type of error is made is the significance level
Type II error - saying something is the same when it is really different (Sloth)
the probability of committing this type of error is designated b
the probability that a comparison procedure will pick up a real difference is called the power of the test and is equal to 1-b
Type I and Type II error rates are inversely related to each other
For a given Type I error rate, the rate of Type II error depends on
sample size
variance
true differences among means

13. Nobody likes to be wrong...
Protection against Type I is choosing a significance level
Protection against Type II is a little harder because
it depends on the true magnitude of the difference which is unknown
choose a test with sufficiently high power
Reasons for not using LSD for more than t-1 comparisons
the chance for a Type I error increases dramatically as the number of treatments increases
for example, with only 20 means - you could make a type I error 95% of the time (in 95/100 experiments)

14. Comparisonwise vs Experimentwise Error
Comparisonwise error rate ( = C)
measures the proportion of all differences that are expected to be declared real when they are not
Experimentwise error rate (E)
the risk of making at least one Type I error among the set (family) of comparisons in the experiment
measures the proportion of experiments in which one or more differences are falsely declared to be significant
the probability of being wrong increases as the number of means being compared increases

15. Comparisonwise vs Experimentwise Error
Experimentwise error rate (E)
Probability of no Type I errors = (1-C)x
where x = number of pairwise comparisons
Max x = t(t-1)/2 , where t=number of treatments
 Probability of at least one Type I error
E = 1- (1-C)x
Comparisonwise error rate
C = 1- (1-E)1/x
if t = 10, Max x = 45, E = 90%

16. Fisher’s protected LSD (FPLSD)
Uses comparisonwise error rate
Computed just like LSD but you don’t use it unless the F for treatments tests significant
So in our example data, any difference between means that is greater than is declared to be significant
LSD = tα 2MSE / r

17. Waller-Duncan Bayes LSD (BLSD)
Do ANOVA and compute F (MST/MSE) with q and f df (corresponds to table nomenclature)
Choose error weight ratio, k
k=100 corresponds to 5% significance level
k=500 for a 1% test
Obtain tb from table (A7 in Petersen)
depends on k, F, q (treatment df) and f (error df)
Compute
Any difference greater than BLSD is significant
Does not provide complete control of experimentwise Type I error
Reduces Type II error
BLSD = t 2MSE/r

18. Duncan’s New Multiple-Range Test
Alpha varies depending on the number of means involved in the test
Alpha
Error Degrees of Freedom
Error Mean Square
Number of Means
Critical Range
Means with the same letter are not significantly different.
Duncan Grouping Mean N variety A A A
B A
B A
B A B B C

19. Student-Newman-Keuls Test (SNK)
Rank the means from high to low
Compute t-1 significant differences, SNKj , using the HSD
Compare the highest and lowest
if less than SNK, no differences are significant
if greater than SNK, compare next highest mean with next lowest using next SNK
Uses experimentwise for the extremes and comparisonwise for adjacent
where j=1,2,..., t-1, k=2,3,...,t
SNK Q
MSE r j =
a,k, /

20. Using SNK with example data:Q
SNK
Mean Yield Variety Rank Yi F D E B A C
= 15 comparisons
18 df for error
se=
= SQRT(120/4) = 5.477
SNK=Q*se

21. Tukey’s honestly significant difference (HSD)
From a table of studentized range values, select a value of Qa which depends on k (the number of means) and v (error df) (Appendix Table VII in Kuehl)
Compute HSD as
For any pair of means, if the difference is greater than HSD, it is significant
Uses an experimentwise error rate
Dunnett’s test is a special case where all treatments are compared to a control
HSD = Q MSE / r

22. Bonferroni Inequality
E  x * C
where x = number of pairwise comparisons
C = E / x
where E = maximum desired experimentwise error rate
Advantages
simple
strict control of Type I error
Disadvantage
very conservative, low power to detect differences

23. False Discovery Rate
Reject H0

24. Most Popular FPLSD test is widely used, and widely abused
BLSD is preferred by some because
It is a single value and therefore easy to use
Larger when F indicates that the means are homogeneous and small when means appear to be heterogeneous
The False Discovery Rate has nice features, but is it widely accepted in the literature?
Tukey’s HSD test
widely accepted and often recommended by statisticians
may be too conservative if Type II error has more serious consequences than Type I error