Accuracy and power of randomization tests in multivariate analysis of variance with vegetation data Valério De Patta Pillar Departamento de Ecologia Universidade Federal do Rio Grande do Sul Porto Alegre, Brazil

Randomization testing: –Became practical with fast microcomputers. –Applicable to most cases analyzed by classical methods. –Applicable to cases not covered by classical methods.

How good is randomization testing? Is it accurate? Is it powerful enough?

Group comparison by randomization testing Choose a test criterion ( ) to compare the groups Permute the data according to the conditions stated by the null hypothesis (Ho) that the groups do not differ Calculate the test criterion  in the random data and compare it to the value found in the observed data. After many iterations, the probability P( o ≥ ) will be the number of iterations with o ≥ divided by the total number of iterations. Reject Ho if P( o ≥ ) is smaller than a threshold (  ) Manly, B. F. J Randomization, Bootstrap and Monte Carlo Methods in Biology. 2 ed. Chapman and Hall.

Randomization test criteria for multivariate comparisons of any number of groups *Pillar, V. D. & Orlóci, L J. Veg. Sci. 7:

An example How common is a Qb ≥ if Ho were true (that the composition is unrelated to group)? Observed squared distance matrix SQ within groups (Qw) = 0/ /2 + ( )/ /2 + 0/1 = Total sum of squares (Qt)= ( )/10 = SQ between groups(Qb) = = Is there a significant effect of N on vegetation composition as defined by these two PFTs?

Reference set under Ho If Ho true, the observation vector in a given sampling unit is independent from the group to which the unit belongs.

A random permutation and corresponding statistics Observed Permuted SQ within groups (Qw o ) = 0/ /2 + ( )/4 + 8/2 + 0/1 = Total sum of squares (Qt)= ( )/10 = SQ between groups(Qb o ) = = Since, < (Qb o < Qb), this iteration adds zero to the frequency of cases in which Qb o ≥ Qb. Permuted squared distance matrix

After random permutations…

Two-factor designs Test criterion: Q b = Q t - Q w is based on the groups defined by the joint states of the factors. Q b is partitioned as Q b = Q b|A + Q b|B + Q b|AB where Q b|A : sum of squares between l a groups according to factor A disregarding factor B Q b|B : sum of squares between l b groups according to factor B disregarding factor A Q b|AB : sum of squares of the interaction AB, obtained by difference. F-ratio = Q b /Q w

Unrestricted permutation in two-factor design

Two-factor Multivariate Analysis of Variance Observed One random permutation Data: Species (57) composition in 8 vegetation units surveyed in two landscape positions (factor A) and two grazing levels (factor B).

After random permutations… Data: Species (57) composition in 8 vegetation units surveyed in two landscape positions (factor A) and two grazing levels (factor B). Unrestricted random permutations. Test criterion F-ratio = Qb/Qw.

Restricted permutations In two-factor (not nested) designs, for testing one factor, permutations may be restricted to occur within the levels of the other factor (Edgington 1987). Restricted permutation within the levels of factor A (for testing factor B): Edgington, E. S Randomization Tests. Marcel Dekker, New York.

Permutations of residuals instead of raw data

Two-factor multivariate analysis of variance by randomization testing for the effects of landscape position and grazing level in natural grassland, southern Brazil (data from Pillar 1986). The data set contains 16 polled community stands by 60 species. Restricted random permutations for testing factors landscape and grazing. Permutation of residuals removing both factors for testing the interaction.

How good is randomization testing in two-factor multivariate analysis of variance?

Simulation of interaction For each case, 1000 data sets were generated, with distribution properties of real vegetation data and subject to multivariate analysis of variance with randomization testing. When factor or interaction effect is set to zero, the proportion of Ho rejection under a given  threshold estimates Type I Error, the probability of wrongly rejecting Ho when it is true. If Type I Error is equal to , the test is exact. When factor or interaction effect > 0, the proportion of Ho rejection estimates the power of the test, which is the one- complement of Type II Error, the probability of not rejecting Ho when it is false.

Simulated data generated with distributional properties of real data Data set: 16 grassland units described by cover of 60 species. Two factors: landscape position (top-convex, concave-lowland) and grazing levels (grazed, ungrazed). Procedure described by Peres-Neto & Olden (2000): 1.Calculate the mean (   ) and the standard deviation (  ij ) for each species vector i within each group j defined by the four factor level combinations; 2.Standardize these vectors for mean equal 0 and standard deviation equal 1, t hij =(x hij -   )/   ; 3.Randomly permute whole stand vectors across groups; 4.Restore the original dispersion within each group by computing new observations s hij = t hij  , defining in this way a data set with the conditions specified by Ho; 5.Apply to the species vectors the corresponding group differences for factor and interaction effects; 6.Perform the randomization tests using 1000 random permutations; 7.Repeat the steps (3) to (6) 1000 times, recording the proportion of Ho rejection. Peres-Neto, P.R. & Olden, J.D Animal Behaviour 61:

With no factor and interaction effects, type I error is not different from 0.05, as expected by using  = Results of power evaluation by data simulation in two-factor MANOVA. The proportion of Ho rejection at  = 0.05 was obtained for 1000 simulated data sets generated on the basis of plant community data with 16 units and 60 species, with increasing difference between the two groups for factor 1, with no interaction. Each factor combination had equal number of units. For each data set a randomization test was run with 1000 iterations. As the effect of factor 1 increases, type I error for factor 2 and interaction are underestimated with unrestricted permutations with Qb and  -ratio, but not with restricted permutations and residuals.

As the effects of both factors increase, type I error for the interaction is underestimated with unrestricted permutations with Qb and  -ratio, but not with residuals.

As the effect of interaction increases, type I error for both factors is underestimated with Qb and  -ratio, un- and restricted permutations. But, main factors should not be considered at all when interaction is present!

As the effects of both factors increase, the power of the test with permutations of raw data is decreased for detecting the interaction when using Qb and  -ratio, but not when permuting residuals.

Results of power evaluation by data simulation in two-way designs. The proportion of Ho rejection at  = 0.05 was obtained for 1000 simulated data sets generated on the basis of plant community data with 60 units and 60 species, with increasing relative difference between the four- group factor combinations. Factor combinations had unequal number of units (11: 31, 12: 9, 21: 9, 22: 11). For each data set a randomization test was run with 1000 iterations.

References Anderson, M.J. and ter Braak, C. 2003, Permutation tests for multi-factorial analysis of variance. Journal of Statistical Computations and Simulations 73: Edgington, E. S Randomization Tests. Marcel Dekker, New York. Manly, B. F. J Randomization, Bootstrap and Monte Carlo Methods in Biology. 2 ed. Chapman and Hall. Peres-Neto, P.R. & Olden, J.D Assessing the robustness of randomization tests: examples from behavioural studies. Animal Behaviour 61: Pillar, V. D. & Orlóci, L On randomization testing in vegetation science: multifactor comparisons of relevé groups. Journal of Vegetation Science 7: Pillar, V. D MULTIV Software for multivariate analysis, randomization tests and bootstrapping. Available (minor version, manual included) at: