Yea, we finally get to do Statistics! Kert Viele Department of Statistics University of Kentucky

Scientific Experimentation Scientific theories make predictions. In a scientific experiment to test a theory, a new situation is created where the theory makes a specific prediction. We try to observe if that prediction comes true. If the prediction does come true, it is evidence in favor of the theory (typically not proof). If the prediction does not occur, the theory must be revised, or perhaps complete discarded.

A simple theory A very simple theory is that “All ravens are black”. This theory only makes predictions about ravens, so the natural thing to do to test this theory is to observe ravens. If you observe black ravens, this confirms the theory (doesn’t proof it though, we’d have to observe all ravens to do that) If you observe a white raven, the theory is proven wrong.

Statistical Theories Statistical hypotheses are hypotheses that discuss probabilities. Saying “this coin is fair” means that in the long run 50% of the flips will be heads. Many theories can be placed in this framework. The hypothesis “all ravens are black”, in statistical terms, is that the probability a raven is black is 100%, and thus the probability a raven is not black is 0%.

If the impossible happens…discard the theory In statistical terms, what happens when we observe a non-black raven? We can go “hmm, what were the odds of that? If the hypothesis were true, that would NEVER happen” Therefore, the hypothesis is NOT true.

With statistics, nothing is impossible! Fundamental idea – if the hypothesis says an event is impossible but it happens anyway, the hypothesis is wrong Unfortunately, with most statistical hypothesis, nothing is impossible. Let’s go back to the hypothesis that a particular coin is fair.

What about unlikely events? Suppose we flip the coin 100 times and observe 98 heads and 2 tails. Most people would be more than a little suspicious of whether the coin is fair. The reasonable source of this suspicion is that observing 98 head and 2 tails is quite unlikely. However, it is NOT impossible.

Back to the drawing board In fact, no outcome is impossible. You can get anywhere from 0 to 100 heads in 100 flips. If our standard was only to disprove a hypothesis when something impossible (according to the hypothesis) occurred, then we would never make any progress with the hypothesis “this coin is fair”. We’d always be in limbo.

The “unlikely” standard Since the “impossibility” standard is unavailable for most statistical hypothesis, we use a weaker standard (the best we have). Impossible = with probability 0 Unlikely = with a small probability (more than 0, but small). For historical reasons, unlikely = 5% typically. We work with “if something unlikely occurs, then the hypothesis is likely wrong”, called “rejecting the hypothesis”.

Summary until now…still awake? The statistical idea of hypothesis tests is based on the idea of concluding an hypothesis is likely wrong when an unlikely event occurs. This is of course weaker than completely disproving a hypothesis when an impossible event occurs, but hey, it’s what we got.

Implementing the “unlikely” standard The standard procedure in a statistical hypothesis test is to “reject” the hypothesis when the data fall in a rejection region. The rejection region is set so that, if the hypothesis is true, the rejection region has a 5% (unlikely) chance of occurring. In most standard situations, the hypothesis translates (use probability theory) to an approximate normal distribution for the data.

Both are 5%, which do you reject?

Reject in the tails Generally, you reject the hypothesis if the observed data is too far away from the hypothesized value (i.e. too far into the tails). For normal distributions, you get your usual “reject if the data is 1.96 standard deviations from the predicted mean”. The 1.96 is often just truncated to 2 for “eyeballing” purposes.

You got a problem with that? You may disagree with 5% as a reasonable definition of “unlikely”. You may use something else, but keep in mind that this is just “the way it is” with respect to many journal publications, etc. The probability has all been worked out in many common situations. Most standard statistical packages allow you to enter data (from EXCEL or another package) and just click a few buttons to get an answer. Just remember, know what the buttons means when you click them!

Normal distributions In most common situation, the hypothesis you are testing implies the data will have a normal (bell-shaped distribution). Normal distributions have two parameters, a mean μ (mu, unless you are from England where it is “moo”) and a variance σ (sigma) μ determines the center of the distribution, while σ determines the spread.

Different Normal Distributions

Small σ’s are good. In scientific context, μ is fixed by the process we are studying (the coin has some probability of landing head, our animals have a certain level of activity). We unfortunately don’t know what μ is, and we can’t control it. σ, on the other hand, is much more controllable. Anything that eliminates noise in our data decreases σ.

Why are small σ’s good? Consider two competing scientific theories, one which states μ=μ 0 and another which states μ≠μ 0. Under the null hypothesis μ=μ 0, we expect a normal distribution to appear, centered at μ 0 The value of σ depends on the inherent noise in the problem AND our experimental design choices. For the example that follows, μ 0 = 10.

Our hypothesis testing procedure Reject in the red area, do not reject in the green area

We can make mistakes… Recall we “reject” the hypothesis if our observed data is too far from the hypothesis (i.e. out in the tails of the null distribution). This is what we called the rejection region However, nothing is impossible in most statistical hypotheses, there is a chance we can make a mistake.

Type I error We fixed the rejection region so that, when the null hypothesis is true, we have a 5% chance of incorrect rejecting the hypothesis. This is called the type I error, or “size” of the test. This of course also means that, when the null hypothesis is true, we have a 95% chance of making the correct decision.

What if the hypothesis is false? If our hypothesis is not true, then the real value of μ is something other than μ 0, and our data comes from a different distribution. We keep the same rejection region. What does this look like?

With μ=14, but with the same rejection region. There’s more red.

Type II error Again, when the null distribution is the right one, we have a 5% chance of making a mistake and a 95% of not making a mistake. When the alternative hypothesis is true, we have different probabilities. For the graph shown here, the probability of rejecting is 26.6%. But wait, when the alternative hypothesis is true, rejecting is a good thing.

Type II error, continued When the alternative hypothesis is true, we want to reject the null hypothesis. The probability of doing this is called the “power” of the test. When the alternative hypothesis is true, the act of not rejecting is called type II error. A good test has low probabilities of both type I and type II error.

Decreasing σ results in a better chance of finding the truth. Suppose we were able to decrease σ, making the normal distributions “thinner”. This changes the rejection region, because we still fix the probability of type I error at 5%. Basically, this results in making the “acceptance region”, the green area, smaller. The increased accuracy lets us “hit a smaller target”, so to speak.

Reminder of the old null distribution

The new improved, smaller σ version

Under the alternative distribution, the data appears in the red more often.

All graphs together for comparison

The effect of decreasing σ When the larger σ is used, the power was 26.6%. When the smaller σ is used, the power increases to 98.0%. Thus the probability of type II error is only 2%. Decreasing σ is a VERY good thing.

How do you decrease σ? Thus, we focus a lot of effort on decreasing σ, which gives us better power and a better chance of making a strong scientific conclusion. So how to we decrease σ? Remove noise from the experiment. There are several ways to do this.

Larger sample sizes are better, but more expensive in many ways. Increase sample size! The larger the sample size, the thinner the normal distributions are. Of course, this goal of thinner normals competes with other demands on our time, our budget, and possible ethical concerns (you don’t want to expose people to a potentially dangerous drug any more than you have to, for example) Never run an experiment with too small a sample size, though, you’ll get NOTHING out of it!

Blocking on known sources of noise. If we know a specific variable will affect our responses, we will often try to control for that variable. A quick example – if you are teaching, students who did well on last year’s standardized tests will tend to do well on this year’s standardized tests, and there is only so much that will change that.

Example You are interesting in investigating the relative effectiveness of two different types of strength training (method A and method B) Your response variable is the amount of weight that can be lifted at the end of the training. You also know the amount each subject can currently lift.

Example continued Presumably people who are stronger now will be stronger later. The issue is by how much. One option is to fit an ANCOVA using the current weight as a continuous covariate. If you are concerned about linearity, then a randomized block design is sensible as well.

Conducting the experiment To block on the current weight, take all the subjects and rank them by the current weight they can lift. Divide the sorted list into block on 2 people each (2 = number of methods of strength training) Within each block, assign one person to method A and one person to method B, at random. This makes it FAR less likely method A or method B will have too much of an abundance of “strong people” in their groups than just random assignment of all subjects to groups A and B.

Moral of the randomization Block for what you know makes a difference. The randomization will likely equalize the groups on what you DON’T know makes a difference. It’s possible there aren’t any other factors, and the randomization doesn’t matter, but it’s easy and it potentially helps. Finding out about another variable later is a bad scene.

Repeated measures Animals vary amongst themselves, either through genetic or environmental factors. Where possible, we would like to control this variation. We do this by applying each treatment to each animal. This is “blocking on animal”. Blocking on animal is often not possible, for example the animal may be changed by the treatment. When you block on animal randomize the order of the treatment so no treatment gets consistently placed first or last in the treatment order.

Control groups Whenever making comparisons between treatments, you must have a group of animals for each potential treatment, including the “current”, or “default” treatment. Without such a group, any difference you observe could be due to the fact the animals were involved in an experiment.