Summer School in Statistics for Astronomers V June 1-6, 2009 Robustness, Nonparametrics and some Inconvenient Truths Tom Hettmansperger Dept. of Statistics Penn State University

Robust methods t-tests and F-test rank tests Least squares Nonparametrics

Some ideas we will explore: Robustness Nonparametric Bootstrap Nonparametric Density Estimation Nonparametric Rank Tests Tests for (non-)Normality The goal: To make you worry or at least think critically about statistical analyses.

Abstract Population Distribution, Model Real World Data Probability and Expectation Statistical Inference

Statistical Model, Population Distribution Research Hypothesis or Question in English Measurement, Exp. Design, Data Collection Translate Res. Hyp. or Quest. into a statement in terms of the model parameters Select a relevant statistic Carry out statistical inference Graphical displays Model criticism Sampling Distributions P-values Significance levels Confidence coefficients State Conclusions and Recommdations in English

Parameters in a population or model Typical Values: mean, median, mode Spread: variance (standard deviation), interquartile range (IQR) Outliers Shape: probability density function (pdf), cumulative distribution function (cdf)

NGC 4382 (n = 59) Research Question: How large are the luminosities in NGC 4382? Measure of luminosity (data below) Traditional model: normal distribution of luminosity Translate Res. Q.: What is the mean luminosity of the population? (Here we use the mean to represent the typical value.) The relevant statistic is the sample mean. orig: no: : Statistical Inference: 95% confidence interval for the mean using a normal approximation to the sampling distribution of the mean.

Variable N N* Mean SE Mean StDev NGC 4382_no NGC 4382_orig NGC 4382_ NGC 4382_ NGC 4382_ Minimum Q1 Median Q

Outliers can have arbitrarily large impact on the sample mean, sample standard deviation, and sample variance. A single outlier can increase the width of the t-confidence interval and inflate the margin of error for the sample mean. Inference can be adversely affected. First Inconvenient Truth: Second Inconvenient Truth: It is bad for a small portion of the data to dictate the results of a statistical analysis.

Third Very Inconvenient Truth: The construction of a 95% confidence interval for the population variance is very sensitive to the shape of the underlying model distribution. The standard interval computed in most statistical packages assumes the model distribution is normal. If this assumption is wrong, the resulting confidence coefficient can typically vary significantly. I am not aware of a stable 95% confidence interval for the population variance.

The ever hopeful statisticians

Robustness: structural and distributional Influence or sensitivity curves: The rate of change in a statistic as an outlier is varied. Breakdown: The smallest fraction of the data that must be altered to carry the statistic beyond any preset bound. We want bounded influence and high breakdown. Structural: We would like to have an estimator and a test statistic that are not overly sensitive to small portions of the data.

Distributional robustness: We want a sampling distribution for the test statistic that is not sensitive to changes or misspecifications in the model or population distribution. This type of robustness provides stable p-values for testing and stable confidence coefficients for confidence intervals.

Message: The sample mean is not structurally robust; whereas, the median is structurally robust. It takes only one observation to move the sample mean anywhere. It takes roughly 50% of the data to move the median. (Breakdown) Sensitivity Curve: SC mean (x) = x SC median (x) = (n+1)x (r) if x < x (r) (n+1)x if x (r) < x < x (r+1) (n+1)x (r+1) if x (r+1) < x when n = 2r

Mean Median x Influence Mean has linear, unbounded influence. Median has bounded influence.

Some good news: The sampling distribution of the sample mean depends only mildly on the population or model distribution. (A Central Limit Theorem effect) Provided our data come from a model with finite variance, for large sample size has an approximate standard normal distribution (mean 0 and variance 1). This means that the sample mean enjoys distributional robustness, at least approximately. We say that the sample mean is asymptotically nonparametric.

More inconvenient truth: the sample variance is neither structurally robust (unbounded sensitivity and breakdown tending to 0), but also lacks distributional robustness. Again, from the Central Limit Theorem: Provided our data come from a model with finite fourth moment, for large sample size has an approximate standard normal distribution with mean 0 and variance:  is called the kurtosis

 Approx true Conf Coeff The kurtosis and is a measure of the tail weight of a model distribution. It is independent of location and scale and has value 3 for any normal model. Assuming 95% confidence:

A very inconvenient truth: A test for normality will also mislead you!!

Some questions: 1.If statistical methodology based on sample means and sample variances is non robust, what can we do? Are you concerned about the last least squares analysis you carried out? (t-tests and F-tests) If not, you should be! 2.What if we want to simply replace the mean by the median as the typical value? The sample median is robust, at least structurally. What about the distribution? 3.The mean and the t-test go together. What test goes with the median?

We know that: How to find SE(median) and estimate it. Two ways: 1.Nonparametric Bootstrap (computational) 2.Estimate the standard deviation of the approximating normal distribution. (theoretical)

Result for NGC4382: SE(median) =.028 (.027 w/o the outlier,.028 w outlier = 24) Nonparametric Bootstrap; 1. Draw a sample of size 59 from the original NGC4382 data. Sample with replacement. 2.Compute and store the sample median. 3.Repeat B times. (I generally take B = 4999) 4.The histogram of the B medians is an estimate of sampling distribution of the sample median. 5.Compute the standard deviation of the B medians. This is the approximate SE of the sample median.

Theoretical (Mathematical Statistics) Moderately Difficult Let M denote the sample median. Provided the density (pdf) of the model distribution is not 0 at the model median, has an approximate normal distribution with mean 0 and variance 1/[4f 2 ( . In other words, SE(median) = and we must estimate the value of the density at the population median. where f(x) is the density and  is the model median.

Let f(x) denote a pdf. Based on a sample of size n we wish to estimate f(x 0 ) where x 0 is given. Define: Where K(t) is called the kernel and Nonparametric density estimation:

Then a bit of calculation yields: And a bit more: And so we want:

The density estimate does not much depend on K(t), the kernel. But it does depend strongly on h, the bandwidth. We often choose a Gaussian (normal) kernel: Next we differentiate the integrated mean squared error and set it equal to 0 to find the optimal bandwidth (indept of x 0 ). If we choose the Gaussian kernel and if f is normal then:

Recall, SE(median) = For NGC4382: n = 59, M = Bootstrap result for NGC4382: SE(median) =.028 finite sample approx Final note: both bootstrap and density estimate are robust.

The median and the sign test (for testing H 0 :  = 0) are related through the L 1 norm. To test H 0 :  = 0 we use S + (  X i > 0 which has a null binomial sampling distribution with parameters n and.5. This test is nonparametric and very robust.

Research Hypothesis: NGC4494 and NGC4392 differ in luminosity. Luminosity measurements (data) NGC 4494 (n = 101) … NGC 4382 (n = 59) … Statistical Model Two normal populations with possibly different means but with the same variance. Translation: H 0 :  4494 =   vs. H 0 :    

Select a statistic: The two sample t statistic The two sided t-test with significanc level.05 rejects the null hyp when |t| > 2. Recall that means and variances are not robust. VERY STRANGE!

Ratio of variances Ratio of sample sizes 1/41/13/1 1/ /1.05 4/ Table of true values of the significance level when the assumed level is.05. Another inconvenient truth: the true significance level can differ from.05 when some model assumptions fail.

An even more inconvenient truth: These problems extend to analysis of variance and regression. Seek alternative tests and estimates. We already have alternatives to the mean and t-test: the median and the sign test that are robust.

We next consider nonparametric rank tests and estimates for comparing two samples. (Competes with the two sample t-test and difference in sample means.) Generally suppose: To test H 0 :  or to estimate  we introduce is the rank of Y j in the combined data.

The robust estimate of  is Provides the robustness Provides the comparison As opposed to which is not robust.

Research Hypothesis: NGC4494 and NGC4392 differ in luminosity. Luminosity measurements (data) NGC 4494 (n = 101) … NGC 4382 (n = 59) … Statistical Model Two normal populations with possibly different medians but with the same scale. Translation: H 0 :   vs. H 0 : 

Mann-Whitney Test and CI: NGC 4494, NGC 4382 N Median NGC NGC Point estimate for Delta is Percent CI for Delta is (-0.328,-0.182) W = (Sum of ranks of NGC 4494) Test of Delta = 0 vs Delta not equal 0 is significant at (P-Value)

Recall the two sample t-test is sensitive to the assumption of equal variances. The Mann-Whitney test is less sensitive to the assumption of equal scale parameters. The null distribution is nonparametric. It does not depend on the common underlying model distribution. It depends on the permutation principle: Under the null hypothesis, all (m+n)! permutations of the data are equally likely. This can be used to estimate the p-value of the test: sample the permutations, compute and store the MW statistics, then find the proportion greater than the observed MW.

Here’s a bad idea: Test the data for normality using, perhaps, the Kolmogorov-Smirnov test. If the test accepts normality then use a t-test, and if it rejects normality then use a rank test. You can use the K-S test to reject normality. The inconvenient truth is that it may accept many possible models, some of which can be very disruptive to the t-test and sample means.

Absolute Magnitude Planetary Nebulae Milky Way Abs Mag (n = 81) …

But don’t be too quick to “accept” normality:

Null Hyp: Pop distribution, F(x) is normal The Kolmogorov-Smirnov Statistic The Anderson-Darling Statistic

A Strategy: Use robust statistical methods whenever possible. If you must use traditional methods (sample means, t and F tests) then carry out a parallel analysis using robust methods and compare the results. Start to worry if they differ substantially. Always explore your data with graphical displays. Attach probability error statements whenever possible.

What more can we do robustly? 1.Multiple regression 2.Analysis of designed experiments (AOV) 3.Analysis of covariance 4.Multivariate analysis These analyses can be carried out using the website:

There’s more: The rank based methods are 95% efficient relative to the least squares methods when the underlying model is normal. They may be much more efficient when the underlying model has heavier tails than a normal distribution. But time is up.

References: 1.Higgins (2004) Intro to Modern Nonpar Stat 2.Hollander and Wolfe (1999) Nonpar Stat Methods 3.Arnold Notes, Bendre Notes 4.Johnson, Morrell, and Schick (1992) Two-Sample Nonparametric Estimation and Confidence Intervals Under Truncation, Biometrics, 48, Staudte and Sheather (1990) Robust Estimation & Testing 6.Efron and Tibshirani (1993) An Introduction to the bootstrap 7.Wasserman (2006) All of Nonparametric Statistics 8.Website:

Thank you for listening!