Computer algebra and rank statistics Alessandro Di Bucchianico HCM Workshop Coimbra November 5, 1997

2 How to run this presentation? the presentation runs itself most of the time click the mouse if you want to continue type S to stop or restart the presentation underlined items are hyperlinks to files on the World Wide Web (usually Postscripts files of technical reports) Enjoy my presentation!

3 Outline General remarks on nonparametric methods What is computer algebra? Case study: the Mann-Whitney statistic Critical values of rank test statistics Moments of the Mann-Whitney statistic Conclusions

4 General remarks on nonparametric methods Practical problems tables (limited, errors, not exact,…) limited availability in statistical software procedures in statistical software often only based on asymptotics

5 General remarks on nonparametric methods Mathematical problems in general no closed expression for distribution function direct enumeration only feasible for small sample sizes recurrences are time-consuming

6 What is computer algebra?

7 Case study: Mann-Whitney statistic independent samples X 1,…,X m and Y 1,…,Y n continuous distribution functions F, G resp. (hence, no ties with probability one) order the pooled sample from small to large

8 Mann-Whitney (continued) Wilcoxon: W m,n =  i rank(X i ) Mann-Whitney: M m,n = #{(i,j) | Y j < X i } W m,n = M m,n + ½ m (m+1) What is the distribution of M m,n under H 0 :F=G?

9 Under H 0, we have:

10

11 Computational speed (Pentium 133 MHz) Exact: P(M 5,5  4) = 1/21  computing time: 0.05 sec (generating function: degree 25) P(M 5,5  4)  Exact: P(M 20,20  138) = (rounded) computing time: 8.5 sec (generating function: degree 400) P(M 20,20  138)  Asymptotics and exact calculations are both useful!

12 Other examples of nonparametric test statistics with closed form for generating function include: Wilcoxon signed rank statistic Kendall rank correlation statistic Kolmogorov one-sample statistic Smirnov two-sample statistic Jonckheere-Terpstra statistic Consult the combinatorial literature! What to do if there is no generating function?

13 Linear rank statistics Z = 1 if th order statistic in the pooled sample is an X-observation, and 0 otherwise Streitberg & Röhmel 1986 (cf. Euler 1748): Branch-and-bound algorithm (Van de Wiel)

14 Moments of Mann-Whitney statistic Mann and Whitney (1947) calculated 4th central moment Fix and Hodges (1955) calculated 6th central moment Computations are based on recurrences Can we improve? solution: computer algebra and generating functions

15 Computing moments of M m,n recompute E(M m,n ) (following René Swarttouw)

16

17 Hence, it remains to calculate for 1  k  m : After some simplifications:

18 L’Hôpital’s rule yields that the limit equals: It is tedious to perform these computations by hand. Alternative: compute moments using Mathematica.

19 Mathematica procedures for moments of M m,n :

20 8th central moment of M m,n

21 Conclusions generating functions are also useful in nonparametric statistics computer algebra is a natural tool for mathematicians asymptotics and exact calculations complement each other

22 Topics under investigation tests for censored data power calculations nonparametric ANOVA (Kruskal-Wallis, block designs, multiple comparisons)block designs Spearman’s  (rank correlation)Spearman’s  multimedia/ World Wide Web implementation Click on underlined items to obtain Postscript file of technical report

23 References A. Di Bucchianico, Combinatorics, computer algebra and the Wilcoxon-Mann- Whitney test, to appear in J. Stat. Plann. Inf.Combinatorics, computer algebra and the Wilcoxon-Mann- Whitney test B. Streitberg and J. Röhmel, Exact distributions for permutation and rank tests: An introduction to some recently published algorithms, Stat. Software Newsletter 12 (1986), 10-18

24 References (continued) M.A. van de Wiel, Exact distributions of nonparametric statistics using computer algebra, Master’s Thesis, TUE, 1996 M.A. van de Wiel and A. Di Bucchianico, The exact distribution of Spearman’s rho, technical report The exact distribution of Spearman’s rho M.A. van de Wiel, A. Di Bucchianico and P. van der Laan, Exact distributions of nonparametric test statistics using computer algebra, technical reportExact distributions of nonparametric test statistics using computer algebra

25 References (continued) M.A. van de Wiel, Edgeworth expansions with exact cumulants for two-sample linear rank statistics, technical reportEdgeworth expansions with exact cumulants for two-sample linear rank statistics M.A. van de Wiel, Exact distributions of two-sample rank statistics and block rank statistics using computer algebra, technical report Exact distributions of two-sample rank statistics and block rank statistics using computer algebra

26 The End