The Idea of the Statistical Test

A statistical test evaluates the "fit" of a hypothesis to a sample.

A statistical hypothesis is a claim regarding a population. The form of a statistical hypothesis usually involves the value(s) of parameters (means, proportions, etc.) associated with a population.

A statistical test compares a statistical hypothesis to a random sample from the relevant population, and indicates the "fit" (good or bad) of the sample and hypothesis. Four basic components are required to perform a statistical test: A random sample meeting assumptions required for the test; A hypothesis to be evaluated - the null hypothesis, or H 0 ; An alternative hypothesis, or H 1 ; A method of checking the fit of the sample to the null hypothesis, or an error function.

The significance of a test is usually given via a p-value; a p-value is the conditional probability of getting a sample error more severe than our sample error, given that the null hypothesis holds. That is, p-value = Pr{getting a worse sample error | H 0 is true}.

A p-value gives the strength of fit of our sample to the null hypothesis. Large p-values indicate strong fit, small p-values indicate weak fit. If the p-value is sufficiently small, we may elect to reject the null hypothesis. In practice, p-values below.05 or.01 are viewed as statistically significant or as highly statistically significant.

Population Family of Samples Error Rule: Sample versus Null Hypothesis Family of Errors Computation of p-value