An Simple Introduction to Statistical Significance Julie Graves MST Conference June 2014

Consider a study evaluating a new weight loss drug. Group A received the drug and lost an average of four kilograms (kg) in seven weeks. Group B didn't receive the drug but still lost an average of one kg over the same period. Did the drug produce this three-kg difference in weight loss? Could some unmeasured influence have led to the three-kg difference in weight loss? Or could it be that Group A lost more weight simply by chance?

It is possible that the drug given to group A did not actually have any influence on weight loss. A statistical analysis could be used to determine how likely it is that the observed difference — in this case, the three-kg difference in average weight loss — might have occurred by chance alone.

Statistical analysis may show that the observed difference has a very small probability of occurring by chance if the treatment did not really have an effect on weight loss. If the probability is small (less than 5%), the researchers would conclude that the observed differences in weight loss should be attributed to the treatment, not to chance.

An observed difference that is unlikely to have occurred by chance is called statistically significant.

Are drivers more distracted when using a cell phone than when talking to a passenger in the car? Researchers wanted to find out, so they designed an experiment. Here are the details:.

In a study involving 48 people, 24 people were randomly assigned to drive in a driving simulator while using a cell phone. The remaining 24 were assigned to drive in the driving simulator while talking to a passenger in the simulator. Part of the driving simulation for both groups involved asking drivers to exit the freeway at a particular exit. In the study, 7 of the 24 cell phone users missed the exit, while 2 of the 24 talking to a passenger missed the exit.

Note that 9 out of 48 drivers missed the exit. Maybe the exit-missers were going to miss the exit regardless of which group they were assigned to. Maybe cell phone or passenger conversations don’t have anything to do with missing exits. Of the 48 drivers in the study, maybe 9 are just inherent exit- missers. If this is the case, we would have expected to see 4 or 5 exit-missers among the 24 cell phone users. The fact that we saw 7 exit-missers among the 24 cell phone users could be just a chance occurrence.

We need to choose between two possibilities Drivers on cell phones are more likely to miss an exit than are other drivers. Drivers on cell phones are not more likely to miss an exit. It was due to chance that we observed 7 exit-missers among the 24 cell phone users.

Statisticians can calculate the probability that an observed difference occurred by chance. Rather than doing calculations, we will use a simulation to estimate this probability. Remove 4 spades from a standard deck of playing cards. The modified deck will contain 13 hearts, 13 clubs, 13 diamonds and 9 spades. The 39 non-spades will represent the drivers who did not miss an exit, and the 9 spades will represent the exit-missers.

Here is how our simulation will work From the 48 cards, randomly select 24 cards to represent the 24 cell phone users. Count how many exit-missers were cell phone users. Repeat. Pool results with other simulators.

Is it unlikely that the group of 24 cell phone users would contain 7or more exit-missers by chance? If the answer is yes, then the difference between 7 exit-missers in the cell phone group and 2 exit missers in the passenger group is statistically significant. If the answer is no, then the difference between 7 exit-missers in the cell phone group and 2 exit missers in the passenger group is not statistically significant.

Suppose that in a clinical trial patients were randomly assigned to either Treatment A or Treatment B. Suppose that so far, Treatment A has a cure rate of 100% and Treatment B has a cure rate of only 50%. That's a pretty dramatic sounding difference. Is it a significant difference?

It is possible that only two patients have been assigned to each treatment and both were cured with treatment A, but only one of the two was cured with treatment B. Treatment A would have a 100% cure rate and B would have a 50% cure rate.

Maybe the next patient assigned to treatment A will not be cured and the next patient assigned to treatment B will be. With three patients assigned to each group, the cure rate is 66% for both treatments. What does all this tell you about the "real" cure rate for each treatment? Can we conclude that the cure rate is the same for both, 66%?

After more patients are assigned to each treatment group, we might observe that Treatment A has a 52% cure rate and B has a 51% cure rate. Researchers would want to know if the observed difference between cure rates is a statistically significant difference.

What do we mean if we say that the difference between two cure rates is statistically significant? We do not mean that the difference is A dramatic difference A large difference An important difference We mean that an observed difference is unlikely to have occurred by chance.