Elementary hypothesis testing Purpose of hypothesis testing Type of hypotheses Type of errors Critical regions Significant levels Power of tests Hypothesis vs intervals R commands for tests

Purpose of hypothesis testing Statistical hypotheses are in general different from scientific ones. Scientific hypotheses deal with the behavior of scientific subjects such as interactions between all particles in the universe. These hypotheses in general cannot be tested statistically. Statistical hypotheses deal with the behavior of observable random variables. These are hypotheses that are testable by observing some set of random variables. They are usually related to the distribution(s) of observed random variables. For example if we have observed two sets of random variables x=(x1,x2,,,,xn) and y=(y1,y2,,,,ym) then one natural question arises: are means of these two sets are different? It is a statistically testable hypothesis. Another question may arise do these two sets of random variables come from the population with the same variance? Or do these distribution come from the populations with the same distribution? These questions can be tested using observed samples.

Types of hypotheses Hypotheses can in general be divided into two categories: a) parametric and b) non-parametric. Parametric hypotheses concern with situations when the distribution of the population is known. Parametric hypotheses concern with the value of one or several parameters of this distribution. Non-parametric hypotheses concern with situations when none of the parameters of the distribution is specified in the statement of the hypothesis. For example hypothesis that two sets of random variables come from the same distribution is non-parametric one. Parametric hypotheses can also be divided into two families: 1) Simple hypotheses are those when all parameters of the distribution are specified. For example hypothesis that set of random variables comes from normal distribution with known variance and known mean is a simple hypothesis 2) Composite hypotheses are those when some parameters of the distribution are specified and others remain unspecified. For example hypothesis that set of random variables comes from the normal distribution with a given mean value but unknown variance is a composite hypothesis.

Errors in hypothesis testing Hypothesis is usually not tested alone. It is tested against some alternative one. Hypothesis being tested is called the null-hypothesis and denoted by H0 and alternative hypothesis is denoted H1. Subscripts may be different and reflect nature of the alternative hypothesis. Null-hypothesis gets “benefit of doubt”. There are two possible conclusions: reject null-hypothesis or not-reject null-hypothesis. H0 is only rejected if sample data contains sufficiently strong evidence that it is not true. Usually testing of hypothesis comes to verification of some test statistic (function of the sample points). If this value belongs to some region w hypothesis is rejected.. This region is called critical region. The region complementary to the critical region that is equal to W-w is called acceptance region. By rejecting or accepting hypothesis we can make two types of errors: Type I error: Reject H0 if it is true Type II error: Accept H0 when it is false. Type I errors usually considered to be more serious than type II errors. Type I errors define significance levels and Type II errors define power of the test. In ideal world we would like to minimize both of these errors.

Power of a test The probability of Type I error is equal to the size of the critical region, . The probability of the type II error is a function of the alternative hypothesis (say H1). This probability usually denoted by . Using notation of probability we can write: Where x is the sample points, w is the critical region and W-w is the acceptance region. If the sample points belongs to the critical region then we reject the null-hypothesis. Above equations are nothing else than Type I and Type II errors written using probabilistic language. Complementary probability of Type II error, 1- is also called the power of the test of the null hypothesis against the alternative hypothesis.  is the probability of accepting null-hypothesis if alternative hypothesis is true and 1- is the probability of rejecting H0 if H1 is true Power of the test is the function of the alternative hypothesis specification of H1. It is usual to use test statistics instead of sample points to define critical region and significance levels.

Critical region Let us assume that we want to test if some parameter of the population is equal to a given value against alternative hypothesis. Then we can write: Test statistic is usually a point estimation for  or somehow related to it. If critical region defined by this hypothesis is an interval (-;cu] then cu is called the critical value. It defines upper limit of the critical interval. All values of the statistic to the left of cu leads to rejection of the null-hypothesis. If the value of the test statistic is to the right of cu this leads to not-rejecting the hypothesis. This type of hypothesis is called left one-sided hypothesis. Problem of the hypothesis testing is either for a given significance level find cu or for a given sample statistic find the observed significance level (p-value).

Significance level It is common in hypothesis testing to set probability of Type I error,  to some values called the significance levels. These levels usually set to 0.1, 0.05 and 0.01. If null hypothesis is true and probability of observing value of the current test statistic is lower than the significance levels then hypothesis is rejected. Consider an example. Let us say we have a sample from the population with normal distribution N(,2). We want to test following null-hypothesis against alternative hypothesis: H0:  = 0 and H1:  < 0 This hypothesis is left one-sided hypothesis. Because all parameters of the distribution (mean and variance of the normal distribution) have been specified it is a simple hypothesis. Natural test statistic for this case is the sample mean. We know that sample mean has normal distribution. Under null-hypothesis mean for this distribution is 0 and standard deviation is /n. Then we can write: If we use the fact that Z is standard normal distribution (mean 0 and variance 1) then using the tables of standard normal distribution we can solve this equation.

Significance level: Cont. Let us define: Then we need to solve the equation (using standard tables or programs): Having found z we can solve the equation w.r.t cu. If the sample mean is less than this value of cu we would reject with significance level . If sample mean is greater than this value then we would not reject null-hypothesis. If we reject (sample mean is smaller than cu) then we would say that if the population mean would be equal to 0 then probability that we would observe sample mean is . To find the power of the test we need to find probability under condition that alternative hypothesis is true.

Significance level: An example. Let us assume that we have a sample of size 25 and sample mean is 128. We know that this sample comes from the population with normal distribution with standard deviation 5.4. We do not know population mean. We want to test the following hypothesis: H0: =130, against H1: <130 We have 0 = 130. Let us set significance level to 0.05. Then from the table we can find that z0.05=1.645 and we can find cu. cu= 0 –z0.05 5.4/25 = 130-1.645 *5.4/5 = 128.22 Since the value of the sample mean (128) belongs to the critical region (I.e. it is less than 128.22) we would reject null-hypothesis with significance level 0.05. Test we performed was left one-sided test. I.e. we wanted to know if the value of the sample mean is less than assumed value (130). Similarly we can build right one-sided tests and combine these two tests and build two sided tests. Right sided tests would look like H0: =0 against H1: >0 Then critical region would consist of interval [cl;). Where cl is the lower bound of the critical region And two sided test would look like H0: =0 against H1: 0 Then critical region would consists combination of two intervals (-;cu] [cl;).

Power of a test Power of a test depends on alternative hypothesis. Power of tests is the probability of critical region under alternative hypothesis. Let us consider the example above. Let us assume that the null hypothesis is that the mean value is 129. Since the mean value of the sample is outside the critical region we do not reject this hypothesis. (129-z0.05*5.4/5 = 127.2 < 128). Let us take an alternative hypothesis: mean is equal 126. We want to know what is the probability of accepting null hypothesis if alternative would be true. Take 0.05 confidence level. Then cu=127.2. We want to find probability of x<cu if alternative hypothesis is true. We want to find: P(x<cu | =126) = 0.86. The power of hypothesis is 0.86. I.e. if the mean value would be 126, by accepting null hypothesis that the mean value is equal to 129 we would make 86% of cases mistakes. Note that if the distribution of the population (therefore each sample point and sample statistics) is known then the power of the test with a given difference could be calculated even before sample is available. Power of the test depends on alternative hypothesis that is the difference between alternative and null hypothesis (1-0, where 0 is mean under null and 1 is under alternative hypothesis), the sample size and standard deviations.

Critical regions and power The table shows schematically relation between relevant probabilities under null and alternative hypothesis. do not reject reject Null hypothesis is true 1-  (Type I error) Null hypothesis is false  (Type II error) 1- 

Composite hypothesis In the above example we assumed that the population variance is known. It was simple hypothesis (all parameters of the normal distribution have been specified). But in real life it is unusual to know the population variance. If population variance is not known the hypothesis becomes composite (hypothesis defines the population mean but population variance is not known). In this case variance is calculated from the sample and it replaces the population variance. Then instead of normal t distribution with n-1 degrees of freedom is used. Value of z is found from the table of the tn-1 distribution. If n (>100) is large then as it can be expected normal distribution very well approximates t distribution. Above example can be easily extended for testing differences between means of two samples. If we have two samples from the population with equal but unknown variances then tests of differences between two means comes to t distribution with (n1+n2-2) degrees of freedom. Where n1 is the size of the first sample and n2 is the size of the second sample. If variances for both population would be known then test statistics for differences between two means has a normal distribution.

P-value of the test Sometimes instead of setting pre-defined significance level, p-value is reported. It is also called observed significance level. Let us analyse it. Let us consider above example when we had a sample of size 25 with the sample mean 128. We assumed that we knew population variance – 5.4. P-value is calculated as follows: We would reject null-hypothesis with significance level 0.05 but we would not reject it with significance level 0.01. If the population mean would be 130 observing 128 or less has the probability equal to 0.0322. In other words if would draw 100 times sample of size 25 then around three times we would observe that mean value is less or equal to 128.

Hypothesis testing vs intervals Some modern authors in statistics think that significance testing is overworked procedure. It does not make much sense once we have observed the sample. Then it is much better to work with confidence intervals. Since we can calculate statistics related with the parameter we want to estimate then we can make inference that where “true” value of the parameter may lie. As we could see in the above example we would reject particular hypothesis with significance level 0.01 but would accept with the significance level 0.05. Testing hypothesis did not say anything about parameter in spite of the fact that we had the sample mean. On the contrary confidence interval at least says where “true” value may be and do we need more experiment to increase our confidence and reduce interval size.

R commands for tests t.test - one, two-sample and paired t-test var.test - test for equality of variances power.t.test - to calculate power of t-test Some other tests. These are nonparametric tests wilcox.test - test for differences between means (works for one, two sample and paired cases) SignRank - for signed rank ks.test - Kolmogorov-Smirnov test for equality of distributions

Further reading Full exposition of hypothesis testing and other statistical tests can be found in: Stuart, A., Ord, JK, and Arnold, S. (1991) Kendall’s advanced Theory of statistics. Volume 2A. Classical Inference and the Linear models. Arnold publisher, London, Sydney, Auckland Box, GEP, Hunter, WG, Hunter, JS (1978) Statistics for experimenters