Nonparametrics.Zip (a compressed version of nonparametrics) Tom Hettmansperger Department of Statistics, Penn State University References: 1.Higgins (2004) Intro to Modern Nonpar Stat 2.Hollander and Wolfe (1999) Nonpar Stat Methods 3.Arnold Notes 4.Johnson, Morrell, and Schick (1992) Two-Sample Nonparametric Estimation and Confidence Intervals Under Truncation, Biometrics, 48, Beers, Flynn, and Gebhardt (1990) Measures of Location and Scale for Velocities in Cluster of Galaxies- A Robust Approach. Astron J, 100, Website:

Robustness and a little philosophy Robustness: Insensitivity to assumptions a.Structural Robustness of Statistical Procedures b.Statistical Robustness of Statistical Procedures

Structural Robustness (Developed in 1970s) Influence: How does a statistical procedure respond to a single outlying observation as it moves farther from the center of the data. Want: Methods with bounded influence. Breakdown Point: What proportion of the data must be contaminated in order to move the procedure beyond any bound. Want: Methods with positive breakdown point. Beers et. al. is an excellent reference.

Statistical Robustness (Developed in 1960s) Hypothesis Tests: Level Robustness in which the significance level is not sensitive to model assumptions. Power Robustness in which the statistical power of a test to detect important alternative hypotheses is not sensitive to model assumptions. Estimators: Variance Robustness in which the variance (precision) of an estimator is not sensitive to model assumptions. Not sensitive to model assumptions means that the property remains good throughout a neighborhood of the assumed model

Examples 1. The sample mean is not structurally robust and is not variance robust. 2.The sample median is structurally robust and is variance robust. 3.The t-test is level robust (asymptotically) but is not structurally robust nor power robust. 4. The sign test is structurally and statistically robust. Caveat: it is not very powerful at the normal model. 5.Trimmed means are structurally and variance robust. 6.The sample variance is not robust, neither structurally nor variance. 7.The interquartile range is structurally robust.

Recall that the sample mean minimizes Replace the quadratic by  x) which does not increase like a quadratic. Then minimize: The result is an M-Estimator which is structurally robust and variance robust. See Beers et. al.

The nonparametric tests described here are often called distribution free because their significance levels do not depend on the underlying model assumption. Hence, they are level robust. They are also power robust and structurally robust. The estimators that are associated with the tests are structurally and variance robust. Opinion: The importance of nonparametric methods resides in their robustness not the distribution free property (level robustness) of nonparametric tests.

Single Sample Methods

Robust Data Summaries Graphical Displays Inference: Confidence Intervals and Hypothesis Tests Location, Spread, Shape CI-Boxplots (notched boxplots) Histograms, dotplots, kernel density estimates.

Absolute Magnitude Planetary Nebulae Milky Way Abs Mag (n = 81) …

But don’t be too quick to “accept” normality:

Null Hyp: Pop distribution, F(x) is normal The Kolmogorov-Smirnov Statistic The Anderson-Darling Statistic

Anatomy of a 95% CI-Boxplot Box formed by quartiles and median IQR (interquartile range) Q3 – Q1 Whiskers extend from the end of the box to the farthest point within 1.5xIQR. For a normal benchmark distribution, IQR=1.348Stdev and 1.5xIQR=2Stdev. Outliers beyond the whiskers are more than 2.7 stdevs from the median. For a normal distribution this should happen about.7% of the time. Pseudo Stdev =.75xIQR

Estimation of scale or variation. Recall the sample standard deviation is not robust. From the boxplot we can find the interquartile range. Define a pseudo standard deviation by.75IQR. This is a consistent estimate of the population standard deviation in a normal population. It is robust. The MAD is even more robust. Define the MAD by Med|x – med(x)|. Further define another psueudo standard deviation by 1.5MAD. Again this is calibrated to a normal population.

No Outlier Break- down Stdev obs.75IQR % 1.5MAD % Suppose we enter 5.14 rather than in the data set. The table shows the effect on the non robust stdev and the robust IQR and MAD.

The confidence interval and hypothesis test

Additional Remarks : The median is a robust measure of location. It is not affected by outliers. It is efficient when the population has heavier tails than a normal population. The sign test is also robust and insensitive to outliers. It is efficient when the tails are heavier than those of a normal population. Similarly for the confidence interval. In addition, the test and the confidence interval are distribution free and do not depend on the shape of the underlying population to determine critical values or confidence coefficients. They are only 64% efficient relative to the mean and t-test when the population is normal. If the population is symmetric then the Wilcoxon Signed Rank statistic can be used, and it is robust against outliers and 95% efficient relative to the t-test.

Two-Sample Methods

Two-Sample Comparisons 85% CI-Boxplots Mann-Whitney-Wilcoxon Rank Sum Statistic Estimate of difference in locations Test of difference in locations Confidence Interval for difference in locations Levene’s Rank Statistic for differences in scale or variance.

Why 85% Confidence Intervals? We have the following test of Rule: reject the null hyp if the 85% confidence intervals do not overlap. The significance level is close to 5% provided the ratio of sample sizes is less than 3.

Mann-Whitney-Wilcoxon Statistic: The sign statistic on the pairwise differences. Unlike the sign test (64% efficiency for normal population, the MWW test has 95.5% efficiency for a normal population. And it is robust against outliers in either sample.

Mann-Whitney Test and CI: App Mag, Abs Mag N Median App Mag (M-31) Abs Mag (MW) Point estimate for d is Percent CI for d is (24.530,25.256) W = Test of d=0 vs d not equal 0 is significant at What is W?

What about spread or scale differences between the two populations? Below we shift the MW observations to the right by 24.9 to line up with M-31. Variable StDev IQR PseudoStdev MW M

Levene’s Rank Test Compute |Y – Med(Y)| and |X – Med(X)|, called absolute deviations. Apply MWW to the absolute deviations. (Rank the absolute deviations) The test rejects equal spreads in the two populations when difference in average ranks of the absolute deviations is too large. Idea: After we have centered the data, then if the null hypothesis of no difference in spreads is true, all permutations of the combined data are roughly equally likely. (Permutation Principle) So randomly select a large set of the permutations say B permutations. Assign the first n to the Y sample and the remaining m to the X sample and compute MMW on the absolute deviations. The approximate p-value is #MMW > original MMW divided by B.

Difference of rank mean abso devs So we easily reject the null hypothesis of no difference in spreads and conclude that the two populations have significantly different spreads.

k-Sample Methods One Sample Methods Two Sample Methods

Variable Mean StDev Median.75IQR Skew Kurtosis Messier Messier NGC NGC NGC All one-sample and two-sample methods can be applied one at a time or two at a time. Plots, summaries, inferences. We begin k-sample methods by asking if the location differences between the NGC nebulae are statistically significant. We will briefly discuss issues of truncation.

Kruskal-Wallis Test on NGC sub N Median Ave Rank Z Overall KW = DF = 2 P = This test can be followed by multiple comparisons. For example, if we assign a family error rate of.09, then we would conduct 3 MWW tests, each at a level of.03. (Bonferroni)

What to do about truncation. 1.See a statistician 2.Read the Johnson, Morrell, and Schick reference. and then see a statistician. Here is the problem: Suppose we want to estimate the difference in locations between two populations: F(x) and G(y) = F(y – d). But (with right truncation at a) the observations come from Suppose d > 0 and so we want to shift the X-sample to the right toward the truncation point. As we shift the Xs, some will pass the truncation point and will be eliminated from the data set. This changes the sample sizes and requires adjustment when computing the corresponding MWW to see if it is equal to its expectation. See the reference for details.

Computation of shift estimate with truncation dmnWE(W) Comparison of NGC4382 and NGC 4494 Data multiplied by 100 and 2600 subtracted. Truncation point taken as 120. Point estimate for d is W = m = 101 and n = 59

Robust regression fitting and correlation (association) Dataset ( We have extracted a sample of 50 from the subset of 2719 Hipparcos stars Vmag = Visual band magnitude. This is an inverted logarithmic measure of brightness Plx = Parallactic angle (mas = milliarcsseconds). 1000/Plx gives the distance in parsecs (pc) B-V = Color of star (mag) The HR diagram logL vs. B-V where (roughly) the log-luminosity in units of solar luminosity is constructed logL=(15 - Vmag - 5logPlx)/2.5. All logs are base10.

Row LogL BV *

The resistant line is robust and not affected by the outliers. It follows the bulk of the data much better than the non robust least squares regression line. There are various ways to find a robust or resistant line. The most typical is to use the ideas of M-estimation and minimize: where the  (x) does not increase as fast as a quadratic. The strength of the relationship between variables is generally measured by correlation. Next we look briefly at non robust and robust measures of correlation or association.

Pearson product moment correlation is not robust. Spearman’s rank correlation coefficient is simply the Pearson coefficient with the data replaced by their ranks. Spearman’s coefficient measures association or the tendency of the two measurements to increase or decrease together. Pearson’s measures the degree to which the measurements cluster along a straight line.

For the example: Pearson r = Spearman r s = Significance tests: Pearson r: refer z to a standard normal distribution, where Spearman r s : refer z to a standard normal distribution, where

Kendall’s tau coefficient is defined as where P is the number of concordant pairs out of n(n-1)/2 total pairs. For example (1, 3) and (2, 7) are concordant since 2>1 and 7>3. Note that Kendall’s tau estimates the probability of concordance minus the probability of discordance in the bivariate population.

For the example: Kendalls Tau = Significance Test: refer z to a standard normal distribution where

What more can we do? 1.Multiple regression 2.Analysis of designed experiments (AOV) 3.Analysis of covariance 4.Multivariate analysis These analyses can be carried out using the website:

Professor Lundquist, in a seminar on compulsive thinkers, illustrates his brain stapling technique.

The End