1. Elementary hypothesis testing
Purpose of hypothesis testing
Type of hypotheses
Type of errors
Critical regions
Significant levels
Power of tests
Hypothesis vs intervals
R commands for tests

2. Example
This example will be used throughout this lecture. This data set was taken from R (it is “shoes” dataset in the package MASS)
The purpose of this experiment was to test wear of shoes of two materials A and B. For this purpose 10 boys were selected and for each boy shoes were made using materials A and B. One type of material was used for shoe for the right foot and another for the left foot. This type of experiment is called paired design. A: B:
For simplicity let us consider difference between A and B vectors:
Sample mean is and sample variance is 0.39

3. Purpose of hypothesis testing
Statistical hypotheses are in general different from scientific ones. Scientific hypotheses deal with the behavior of scientific subjects such as interactions between all particles in the universe. These hypotheses in general cannot be tested statistically. Statistical hypotheses deal with the behavior of observable random variables. These are hypotheses that are testable by observing some set of random variables. They are usually related to the distribution(s) of observed random variables.
For example if we have observed two sets of random variables x=(x1,x2,,,,xn) and y=(y1,y2,,,,ym) then one natural question is: are means of these two sets different? It is a statistically testable hypothesis. Another question may arise do these two sets of random variables come from the population with the same variance? Or do these random variables come from the populations with the same distribution? These questions can be tested using observed samples.

4. Types of hypotheses
Hypotheses can in general be divided into two categories: a) parametric and b) non-parametric. Parametric hypotheses concern with situations when the distribution of the population is known. Parametric hypotheses depend on the value of one or several parameters of this distribution. Non-parametric hypotheses concern with situations when none of the parameters of the distribution is specified in the statement of the hypothesis. For example hypothesis that two sets of random variables come from the same distribution is non-parametric one.
Parametric hypotheses can also be divided into two families: 1) Simple hypotheses are those when all parameters of the distribution are specified. For example hypothesis that set of random variables comes from a population with normal distribution with known variance and known mean is a simple hypothesis 2) Composite hypotheses are those when some parameters of the distribution are specified and others remain unspecified. For example hypothesis that set of random variables comes from a population with normal distribution with a given mean value but unknown variance is a composite hypothesis.

5. Errors in hypothesis testing
Hypothesis is usually not tested alone. It is tested against some alternative one. Hypothesis being tested is called the null-hypothesis and denoted by H0 and alternative hypothesis is denoted H1. Subscripts may be different and may reflect the nature of the alternative hypothesis. Null-hypothesis gets “benefit of doubt”. There are two possible conclusions: reject null-hypothesis or not-reject null-hypothesis. H0 is only rejected if sample data contains sufficiently strong evidence that it is not true. Usually testing of a hypothesis comes to verification of some test statistic (a function of the sample points). If this value belongs to some region w hypothesis is rejected.. This region is called critical region. The region complementary to the critical region that is equal to W-w is called acceptance region. By rejecting or accepting hypothesis we can make two types of errors:
Type I error: Reject H0 if it is true
Type II error: Accept H0 when it is false.
Type I errors usually considered to be more serious than type II errors.
Type I errors define significance levels and Type II errors define power of the test. In ideal world we would like to minimize both of these errors.

6. Power of a test
The probability of Type I error is equal to the size of the critical region, . The probability of the type II error is a function of the alternative hypothesis (say H1). This probability usually denoted by . Using notation of probability we can write:
Where x is the sample points, w is the critical region and W-w is the acceptance region. If the sample points belong to the critical region then we reject the null-hypothesis. Above equations are nothing else than Type I and Type II errors written using probabilistic language.
Complementary probability of Type II error, 1- is called the power of the test of the null hypothesis against the alternative hypothesis.  is the probability of accepting null-hypothesis if alternative hypothesis is true and 1- is the probability of rejecting H0 if H1 is true
Power of the test is the function of , the alternative hypothesis - H1 and probability distributions conditional on H0 and H1.

7. Critical region
Let us assume that we want to test: if some parameter of the population is equal to a given value against alternative hypothesis. Then we can write (for example):
Test statistic is usually a point estimation for  or somehow related to it. If critical region defined by this hypothesis is an interval (-;cu] then cu is called the critical value. It defines upper limit of the critical interval. All values of the statistic to the left of cu leads to rejection of the null-hypothesis. If the value of the test statistic is to the right of cu this leads to not-rejecting the hypothesis. This type of hypothesis is called left one-sided hypothesis. Problem of the hypothesis testing is either for a given significance level find cu or for a given sample statistic find the observed significance level (p-value).

8. Significance level
It is common in hypothesis testing to set probability of Type I error,  to some values called the significance levels. These levels usually set to 0.1, 0.05 and If null hypothesis is true and probability of observing value of the current test statistic is lower than the significance levels then hypothesis is rejected.
Consider an example. Let us say we have a sample from the population with normal distribution N(,2). We want to test following null-hypothesis against alternative hypothesis:
H0:  = 0 and H1:  < 0
This hypothesis is left one-sided hypothesis. Because all parameters of the distribution (mean and variance of the normal distribution) have been specified it is a simple hypothesis. Natural test statistic for this case is the sample mean. We know that sample mean has normal distribution. Under null-hypothesis mean for this distribution is 0 and standard deviation is /n. Then we can write:
If we use the fact that Z is standard normal distribution (mean equal to 0 and variance equal to 1) then using the tables of standard normal distribution we can solve this equation.

9. Significance level: Cont.
Let us define:
Then we need to solve the equation (using standard tables or programs):
Having found z we can solve the equation w.r.t cu.
If the sample mean is less than this value of cu we would reject with significance level . If sample mean is greater than this value then we would not reject null-hypothesis. If we reject (sample mean is smaller than cu) then we would say that if the population mean would be equal to 0 then probability that we would observe sample mean is .
To find the power of the test we need to find probability under condition that alternative hypothesis is true.

10. Significance level: An example.
Let us have a look example we had in the beginning of the lecture. Consider differences between A and B. The sample has a size of 10 and sample mean is We assume that this sample comes from the population with normal distribution with standard deviation We want to test the following hypothesis:
H0: =0, against H1: <0
We have 0 = 0. Let us set significance level to Then from the table we can find that z0.05=1.645 and we can find cu.
cu= 0 –z /10 = *0.39/3.16 = -0.2
Since the value of the sample mean (-0.41) belongs to the critical region (I.e. it is less than -0.2) we would reject null-hypothesis with significance level 0.05 (as well as at the level 0.01).
Note that we could have used R function to get the same result:
qnorm(0.05,sd=0.39/sqrt(10))
Test we have performed was left one-sided test. I.e. we wanted to know if the value of the sample mean is less than assumed value (0). Similarly we can build right one-sided tests and combine these two tests and build two sided tests. Right sided tests would look like
H0: =0 against H1: >0
Then critical region would consist of interval [cl;). Where cl is the lower bound of the critical region
And two sided test would look like
H0: =0 against H1: 0
Then critical region would consists combination of two intervals (-;cu] [cl;).

11. Power of a test Power of a test depends on the alternative hypothesis.
To see the power of the current test let us use normal distribution. Null hypotheis: that true mean is equal 0. Let us set alternative hypothesis: true mean is the observed difference that is equal to Again we assume that the probability distribution is normal with known sd = Sample size is 10. Significance level we want to test is 0.05 as in the previous case. We found that the upper bound of the critical region is To find the power of the test we need to find P(x<-0.2 | 1=-0.41,=0.39). We can use R function to calculate this:
pnorm(-0.2,mean=-0.41,sd=0.39/sqrt(10))
We divided standard deviation by the square root of 10 because of the sample size. This test is very powerful - power is equal 0.96
Again if we do not know the standard deviation of the population then t distribution is more appropriate (it is implemented in the command - power.t.test).
Note that if the distribution of the population (therefore each sample point and sample statistics) is known then the power of the test with a given difference could be calculated even before sample is available. Power of the test depends on alternative hypothesis. In case of mean of normal distribution that is exprssed using the difference between alternative and null hypothesis (1-0, where 0 is mean under null and 1 is under alternative hypothesis), the sample size and standard deviations.

12. Power of test
Power of a test can be used before as well as after experimental data have been collected. Before the experiment it is performed to find out the sample size to detect a given effect. It can be used as a part of the design of an experiment.
After the experiment it uses the sample size, effect (e..g. observed difference between means), standard deviation and calculates the power of the test.
For example if we want to detect difference between means equal to 1 (delta) in paired design with power equal 0.9 at a significance level 0.05 in one sided test then we need around 10 observations.
It is done in R using the command power.t.test(delta=1,sd=1,power=0.8,type='paired',alt='one.sided')
The result of R function:
Paired t test power calculation
n =
delta = 1
sd = 1
sig.level = 0.05
power = 0.8
alternative = one.sided

13. Critical regions and power
The table shows schematically relation between relevant probabilities under null and alternative hypothesis.
do not reject
reject
Null hypothesis is true
1-
 (Type I error)
Null hypothesis is false
 (Type II error)
1- 

14. Composite hypothesis
In the above example we assumed that the population variance is known. It was simple hypothesis (all parameters of the normal distribution have been specified). But in real life it is unusual to know the population variance. If population variance is not known the hypothesis becomes composite (hypothesis defines the population mean but population variance is not known). In this case variance is calculated from the sample and it replaces the population variance. Then instead of normal t distribution with n-1 degrees of freedom is used. Value of z is found from the table of the tn-1 distribution. If n (>100) is large then as it can be expected normal distribution very well approximates t distribution.
Above example can be easily extended for testing differences between means of two samples. If we have two samples from the population with equal but unknown variances then tests of differences between two means comes to t distribution with (n1+n2-2) degrees of freedom. Where n1 is the size of the first sample and n2 is the size of the second sample.
If variances for both population would be known then test statistics for differences between two means has a normal distribution.

15. P-value of the test
Usually instead of setting pre-defined significance level, observed p-value is reported. It is also called observed significance level. Let us analyse it. Let us consider above example when we had a sample of size 10 with the sample mean We assumed that we knew population variance – P-value is calculated as follows:
We would reject null-hypothesis with significance level 0.05, 0.01 etc. If the population mean would be 0 with standard deviation 0.39 then observing or less has the probability equal to In other words if would draw times sample of size of 10 then around four times we would observe that mean value is less or equal to
In this example we assumed that we know the variance of the population. That is why we used normal distribution. If we do not know the variance we should use t (with degree of freedom equal to 10-1=9) distribution and p-value becomes

16. Likelihood ratio test
Likelihood ratio test is one of the techniques to calculate test statistics. Let us assume that we have a sample of size n (x=(x1,,,,xn)) and we want to estimate a parameter vector =( 1,2). Both 1 and 2 can also be vectors. We want to test null-hypothesis against alternative one:
Let us assume that likelihood function is L(x| ). Then likelihood ratio test works as follows: 1) Maximise the likelihood function under null-hypothesis (I.e. fix parameter(s) 1 equal to 10 , find the value of likelihood at the maximum, 2)maximise the likelihood under alternative hypothesis (I.e. unconditional maximisation), find the value of the likelihood at the maximum, then find the ratio:
w is the likelihood ratio statistic. Tests carried out using this statistic are called likelihood ratio tests. In this case it is clear that:
If the value of w is small then null-hypothesis is rejected. If g(w) is the the density of the distribution for w then critical region can be calculated using:

17. Hypothesis testing vs intervals
Some modern authors in statistics think that significance testing is overworked procedure. It does not make much sense once we have observed the sample. Then it is much better to work with confidence intervals. Since we can calculate statistics related with the parameter we want to estimate then we can make inference that where “true” value of the parameter may lie.
R commands produce predefined confidence intervals as well as p values. Usually p-values are used in rejecting or not-rejecting a hypothesis.

18. R commands for tests t.test - one, two-sample and paired t-test
var.test - test for equality of variances
power.t.test - to calculate power of t-test
Some other tests. These are nonparametric tests
wilcox.test - test for differences between means (works for one, two sample and paired cases)
ks.test - Kolmogorov-Smirnov test for equality of distributions

19. Further reading
Full exposition of hypothesis testing and other statistical tests can be found in:
Stuart, A., Ord, JK, and Arnold, S. (1991) Kendall’s advanced Theory of statistics. Volume 2A. Classical Inference and the Linear models. Arnold publisher, London, Sydney, Auckland
Box, GEP, Hunter, WG, Hunter, JS (1978) Statistics for experimenters
Peter Dalgaard, (2008) Introductory statistics with R