P-values and their limitations & Type I and Type II errors Stats Club 8 Marnie Brennan

References Petrie and Sabin - Medical Statistics at a Glance: Chapter 17 & 18 Good Petrie and Watson - Statistics for Veterinary and Animal Science: Chapter 6 Good Kirkwood and Sterne – Essential Medical Statistics: Chapter 8 & 35 Dohoo, Martin and Stryhn – Veterinary Epidemiologic Research: Chapter 2 & 6

What do you know about P-values?

Interesting reads! Sterne, JAC and Davey-Smith, G (2001) Sifting the evidence – what’s wrong with significance tests? British Medical Journal, Vol. 322, Good Altman, DG and Bland, JM (1995) Absence of evidence is not evidence of absence. British Medical Journal, Vol. 311, 485. Nakagawa, S and Cuthill, IC (2007) Effect size, confidence interval and statistical significance: a practical guide for biologists. Biological Reviews, Vol. 82, – I’ve not read this, but it has been recommended

Differences between groups Many different tests to measure the difference between two or more groups of subjects/animals/patients – We will cover these individually in subsequent weeks How do we know whether they are truly different from each other? – i.e. Is there truly a difference between groups or not?

Hypothesis (significance) testing You have a scientific question you want to answer You construct a hypothesis to test your question You have to have an alternative hypothesis to test it against This differs from projects which might be hypothesis generating – i.e. you are exploring possible factors and don’t know yet which ones are more important

Null and alternative hypotheses Null hypothesis – No difference between groups/no association between variables – Sometimes written as H 0 Alternative hypothesis – There is a difference between groups/an association between variables – Sometimes written as H 1 These hypotheses relate to the population of interest, not your sample of the population

Thanks to the-scientific-process/ for the example!!! the-scientific-process/ Where have all my socks gone? An example! Aliens have come to Earth specifically to take my socks There is another reason for the missing socks X NULL HYPOTHESIS ALTERNATIVE HYPOTHESIS

What is a P-value? We do our study, run our statistical tests, and come up with a P-value or Probability ‘The P-value is the probability of obtaining our results or something more extreme, if the null hypothesis is true’ (Petrie and Sabin) ‘The probability of getting a difference at least as big as that observed if the null hypothesis is true’ (Kirkwood and Sterne) ‘The chance of getting the observed effect (or one or more extreme) if the null hypothesis is true’ (Petrie and Watson)

What does this mean??!! Basically the probability of getting what you have got with your study results if the null hypothesis is true! – If the difference between our groups is large The probability would be small, therefore unlikely the null hypothesis is true (and you usually reject the null hypothesis as there is evidence against it) – If the difference between our groups is small The probability would be large, therefore likely the null hypothesis is true (there is not enough evidence to reject the null hypothesis) Bad to say you accept the null hypothesis! – ‘Absence of evidence is not evidence of absence’

A value of the test statistic which gives P>0.05 A value of the test statistic which gives P<0.05 Significant at the 5% level Not significant at the 5% level

Using P-values Usually you set your ‘significance’ level before you collect your data – this should be stated in the methods – e.g. ‘We set the significance level at P<0.01 for our analysis’ P<0.05 is a fairly arbitrary level (one guy’s ponderings!) – Read the article by Sterne and Davey Smith – Bottom line - the smaller the P-value, the more evidence against the null hypothesis

A sliding scale

How does this fit with what you do or have seen/experienced?

P-value etiquette (variable!)! Always quote the exact P-value if you can – E.g. P = 0.032, not P<0.05 Display P-values accurate to two significant figures – E.g. P=0.032, or 0.17 When P-values become very small, acceptable to display as P<0.001

Limitations of using just P-values By just using P-values, you lose a lot of information – Doesn’t tell you about the magnitude of the effect observed Often researchers only talk about P-values, and nothing else – I am certainly guilty of this! It is also important to determine whether your result is biologically or clinically important (not only that it is ‘significant’) – if you just use a number to interpret outcomes, it may not ‘mean’ anything You can use Confidence Intervals (CI’s) to quantify the effect of interest – Gives you a range of values which represent the difference between your groups – There is another Stats Club session on these coming up

Interpretation of research

You Tube video on P-values KLw KLw

Errors in hypothesis testing The rejection of the null hypothesis, or not, can be wrong in studies sometimes Petrie and Watson

Type I error When we reject the null hypothesis and it is actually true – Affected by: Significance level chosen (becomes the maximum chance of making a Type I error) – If significance level P< in 20 chance that a test will be significant by chance – If P< in 100 chance the test is significant by chance Number of comparisons – the greater number of comparisons carried out, the more likely you will get a ‘positive’ result that is spurious (multiple testing issue) – Comes back to whether the result is biologically or clinically important – Can adjust for this using post-hoc analysis e.g. Bonferroni correction

Type II error We don’t reject the null hypothesis when there is evidence to do so – Affected by: Small sample sizes – more chance of getting Type II errors Precision of the measurements – if measurements are precise, less chance of getting Type II errors Effect of interest – the larger the difference between the groups, the less likely that a Type II error will occur

Type I and Type II error - relationship These two things are related, generally as one increases, the other decreases Bottom line – if your study design is correct, you have carried out a sample size calculation and have recruited the right number of subjects, then the chances of error decrease hugely as the power of your study will be sufficient – Sample size calculations and power will be discussed in later Stats Club sessions

Summary Set your significance level BEFORE you start your data collection, and don’t just go automatically for P<0.05 – think about what you are trying to show with your research Display your P-values correctly Use P-values but also confidence intervals to get an idea of the magnitude of the difference between groups Set your study up right to decrease the chances of Type I and Type II errors

My kind of hypothesis testing!!!

Next month Confidence intervals beware……