1. Lecture 2: Some general comments on statistical tests and statistical inference
Elements of a statistical test
Statistical null hypotheses
The meaning of p
Inference: How to translate p into a conclusion
Statistical errors in hypothesis testing
Power
One versus two-tailed tests
Problems with statistical hypothesis testing
Bio 4118 Applied Biostatistics
2001

2. Elements of a statistical test
Null hypothesis (H0)
Observations (data)
Test statistic
Assumptions
Bio 4118 Applied Biostatistics
2001

3. Statistical null hypotheses
The “default” to which you compare your data
Usually, one sets up the analysis such that if you reject the null hypothesis, you have a pattern which is consistent with the biological prediction…
…so that in many cases, the null hypothesis specifies a lack of pattern.
Bio 4118 Applied Biostatistics
2001

4. Biological questions versus statistical null hypotheses
Do males and females differ in size?
Average size of males and females are equal.
Does age structure differ between two fish populations?
Age (frequency) distribution is independent of population.
Bio 4118 Applied Biostatistics
2001

5. Testing biological hypotheses
Biological question
Pose biological question (Do drugs A and B differ in their efficacy in treating breast cancer?)
Generate biological hypothesis
Generate biological prediction
Induction
Biological hypothesis
Deduction
Survival A B
Bio 4118 Applied Biostatistics
2001

6. Observations
Assume that the experimental data (observations) are collected in a proper manner.
All observations are subject to a certain amount of measurement error!
Bio 4118 Applied Biostatistics
2001

7. Statistical analysis as model building
All statistical analyses begin with a mathematical model that supposedly “describes” the data, e.g., regression, ANOVA.
“Model fitting” is then the process by which model parameters are estimated. X
Linear regression
ANOVA
e42 m2 a2 Y m
Group 1
Group 2
Group 3
Bio 4118 Applied Biostatistics
2001

8. Parameters, statistics and estimators
Population
parameters characterize populations (which in general cannot be completely enumerated)
statistics (estimators) are estimates of population parameters obtained from a finite sample (e.g., the sample mean is an estimate of the population mean)
Sample
The process by which one obtains an estimate of a population parameter from a finite sample is called an estimation procedure.
Bio 4118 Applied Biostatistics
2001

9. The meaning of p
Informal: the probability that the null hypothesis is true
Strictly correct: the probability of observing data as deviant (from the expected results) as the observed results if in fact the null hypothesis were true, assuming the data were properly collected, and all statistical assumptions are met.
Bio 4118 Applied Biostatistics
2001

10. To reject or not reject?
The decision to reject or accept the null hypothesis is based on p.
This requires some agreement (convention) as to what p value we will consider as significant. This threshold value is arbitrary!
Bio 4118 Applied Biostatistics
2001

11. Test statistics
In standard statistical analysis, p is estimated by reference to the distribution of an appropriate test statistic.
If we know the distribution of the test statistic, we can calculate the probability of getting a test statistic value at least as large (small) as the calculated value if H0 were true, i.e., p.
Bio 4118 Applied Biostatistics
2001

12. An example
Two samples (1, 2) with mean values that differ by some amount d.
What is the probability p of observing this difference under H0 that the two means are in fact equal?
Sample 2
Sample 1
Frequency
Bio 4118 Applied Biostatistics
2001

13. An example (cont’d)
Frequency
Sample 2
Sample 1
If H0 is true, the expected distribution of the test statistic t is:
Probability (p) t -3 -2 -1 1 2 3
Bio 4118 Applied Biostatistics
2001

14. An example (cont’d) For the two populations, suppose t = 2.01
Frequency
Sample 2
Sample 1
For the two populations, suppose t = 2.01
What is the probability of getting a value at least this large under H0 that the two means are in fact equal?
Since p is small, it is unlikely that H0 is true.
Therefore, reject H0. -3 -2 -1 1 2 3
Probability
t = 2.01
Bio 4118 Applied Biostatistics
2001

15. Inference: How to translate p into a conclusion?
If p < 0.05, reject the null hypothesis...
...but keep p in mind!
Report p, not just whether it is “significant” (or not).
Remember, the p < 0.05 “convention” is entirely arbitrary!
Bio 4118 Applied Biostatistics
2001

16. “Statistical significance” and real-world decision-making: an example
If you were offered the same odds on each horse, on which would you bet?
If you were a bookie, would you offer the same odds on each horse? And if you did, would you still be in business?
Clyde’s Fancy
Hypattia
Bio 4118 Applied Biostatistics
2001

17. Statistical errors in hypothesis testing
Two types: a true null hypothesis may be rejected, or a false null hypothesis may be accepted
Type I error (a): the probability of rejecting a true null hypothesis
Type II error (b) : the probability of accepting a false null hypothesis
Bio 4118 Applied Biostatistics
2001

18. Errors in inference Reality Conclusion H0 is true H0 is false
Accept H0
no error 
Reject H0 
no error
Bio 4118 Applied Biostatistics
2001

19. Errors in inference: an example
Reality
Conclusion
No HIV
HIV
Seronegative
99% 5% 
Seropositive 1% 
95% H0 HA
Bio 4118 Applied Biostatistics
2001

20. One- and two-tailed null hypotheses-3 -2 -1 1 2 3
Probability
a/2
1- a
a/2
For 2-tailed H0, there are two rejection regions of size a/2.
For 1-tailed H0 there is one rejection region of size a. -3 -2 -1 1 2 3 -3 -2 -1 1 2 3
Probability
1- a a a
1- a t
Bio 4118 Applied Biostatistics
2001

21. Example: 2-tailed H0 No difference in populations H0: m1 = m2
Frequency
Sample 2
Sample 1
No difference in populations
H0: m1 = m2
Since H0 is 2- tailed, would reject H0 if m1 - m2 > 0 or m1 - m2 < 0. -3 -2 -1 1 2 3
Probability
Bio 4118 Applied Biostatistics
2001

22. Example: 1-tailed H0
Frequency
Sample 2
Sample 1
The average size of individuals in population 1 is greater than population 2
H0: m1 - m2  0
Since H0 is 1- tailed, would reject H0 if m1 - m2 > 0 only. -3 -2 -1 1 2 3
Probability
Bio 4118 Applied Biostatistics
2001

23. One versus two-tailed hypotheses
Sample 2
Frequency
Sample 1
2-tailed hypothesis: reject if any non-random pattern is detected.
1-tailed hypothesis: reject if a specified directional non-random pattern is detected
H0: m1 = m2 (2-tailed, reject)
H0: m1  m2 (1-tailed, accept)
Bio 4118 Applied Biostatistics
2001

24. Important note! -3 -2 -1 1 2 3
Probability
For given “directionality”, 1- tailed test is more powerful than 2-tailed
Therefore, always specify the nature of H0 before your analysis! a
a/2
Probability 2 3
Bio 4118 Applied Biostatistics
2001

25. Parameters of statistical inference
Type I error rate (a)
Power (1 - Type II error rate = 1 - b)
Sample size (N)
Effect size (d)
Each of the above is a function of the other three. Hence, if three are known, so is the fourth.
Bio 4118 Applied Biostatistics
2001

26. Power
Power is the probability of rejecting the null hypothesis when it is false and a specified alternate null hypothesis is true, i.e. 1- b.
Power can only be calculated when a specific alternate null hypothesis is specified. Therefore, power depends on the alternate null hypothesis.
Powerful tests can detect small differences, weak tests only large differences.
Bio 4118 Applied Biostatistics
2001

27. Calculating power: an example
Expected distribution of means of samples of 5 housefly wing lengths from normal populations specified by m as shown above curves and sY = Centre curve represents null hypothesis, H0: m = 45.5, curves at sides represent alternative hypotheses, m = 37 or m = 54. Vertical lines delimit 5% rejection regions for the null hypothesis.
H1 : m = 37
H0 : m = 45.5
H1 : m = 54
Bio 4118 Applied Biostatistics
2001

28. Power: cont’d
H0: m = m0
H1: m = m1
m1=54
m1=53
m1=50
m1=48.5
b=0.0096
b=0.0018
m0=45.5
b=0.2676
b=0.5948
Increases in type II error, b, as alternative hypothesis, H1, approaches null hypothesis, H0 -- that is, m1 approaches m . Shading represents b. Vertical lines mark off 5% critical regions (2.5% in each tail) for the null hypothesis. To simplify the graph, the alternative distributions are shown for one tail only.
Bio 4118 Applied Biostatistics
2001

29. Effect size
Every null hypothesis in any statistical test implies a value for some population parameter.
E.g. if two sample means are equal, the absolute value of the difference d between the two populations is zero:
Sample 2
Sample 1
Frequency X
Bio 4118 Applied Biostatistics
2001

30. Effect size (cont’d)
Sample 2
Sample 1
Frequency
More generally, since H0 specifies a lack of some phenomenon, d quantifies the degree to which the phenomenon is present.
So if H0 is false, it is false to some specific degree, quantified by d, the effect size. X
Bio 4118 Applied Biostatistics
2001

31. Types of power analysis I: power as a function of a, d and N
Often done after a statistical test, where N (sample size) and effect size (d) are determined and the null hypothesis has been accepted.
Then, for specified a, we can calculate 1- b (the power of the test)
If 1- b is low, then the Type II error rate is large, so there is a good chance we have accepted a false H0.
Sample 2
Sample 1
Frequency X
Bio 4118 Applied Biostatistics
2001

32. Types of power analysis II: N as a function of a, d and power
A certain effect size (d) is anticipated (perhaps based on a preliminary sample) with a desired a and 1- b.
Given a, b and d, we can calculate the minimum sample size Nmin required to achieve the desired specifications.
This exercise can be very useful in planning experiments.
Pre-sample 2
Pre-sample 1
Frequency X
Bio 4118 Applied Biostatistics
2001

33. Types of power analysis III: d as a function of a, N and power
Given a desired a, 1- b and N, what is the minimal detectable effect size dmin?
If dmin is large, then only large deviations from H0 will be detected (i.e. will result in rejection of H0).
Thus, we should be VERY VERY careful NOT to infer that some phenomenon does not exist if we accept H0.
Sample 2
Sample 1
Frequency X
Bio 4118 Applied Biostatistics
2001

34. Power: dependence on sample size
1.0
0.5 a
Wing length (x 0.1 mm)
Power (1-b)
n = 5
n = 35
Power curves for testing H0: m = H1: m  45.5 for n = 5 and for n = 35.
For given observed wing length, the probability of rejecting a false null hypothesis decreases as N decreases.
Bio 4118 Applied Biostatistics
2001

35. Why power matters
N = 200
Frequency
Two samples, identical means and variances, but differ in N
in first case, power is large, p < .05, therefore reject H0
in second case, power is low, p > .05, therefore accept H0.
N = 30
Frequency m1 m2
Size
Bio 4118 Applied Biostatistics
2001

36. Power: conclusions
If sample sizes are small, the power of any test is usually low.
So, unless one knows the power of the analysis, a decision to accept the null hypothesis is meaningless!
Conversely, if power is very high, rejection of the null is very likely, even if deviations from null expectations are small (and perhaps biologically meaningless)!
Bio 4118 Applied Biostatistics
2001

37. Statistical hypothesis testing: problems and caveats
Problem 1: many H0s are very unlikely to be true a priori…
…so that their rejection is not very informative.
Treatment 1
Treatment 2
Control
Average yield
Treatment
Bio 4118 Applied Biostatistics
2001

38. Statistical hypothesis testing: problems and caveats
Problem 2: Nominal type I error (e.g. a = 0.5) is entirely arbitrary, and may not bear any relationship to biological significance…
… and even less to decision-making
Threshold for
decision-making
Probabilty -3 -2 -1 1 2 3 t
Bio 4118 Applied Biostatistics
2001

39. Statistical hypothesis testing: problems and caveats
Problem 3: p is probability of obtaining a test statistic at least as extreme as that observed if H0 is true…
… but often the actual (sampling) distribution of the test statistic does not match the (assumed) distribution under the null.
Sampled
Probabilty
Null -3 -2 -1 1 2 3 t
Bio 4118 Applied Biostatistics
2001

40. Statistical hypothesis testing: problems and caveats
Problem 4: for fixed effect size, p depends on sample size (n)…
…so that one can almost always reject H0 if the sample is sufficiently large, even if the observed effect is trivial
Larger effect size
Type I error
Smaller effect size
0.05
Sample size (n)
Bio 4118 Applied Biostatistics
2001

41. Statistical hypothesis testing: problems and caveats
Problem 5: since p depends on sample size (n)…
… using a fixed nominal a (e.g. a = 0.05) as n increases is logically inconsistent: even for n = infinity and true H0, a = 0.05!
Fixed a (e.g. 0.05)
0.05
a depends on n
Nominal type I error (a)
Sample size (n)
Bio 4118 Applied Biostatistics
2001

42. Statistical hypothesis testing: solutions
Avoid testing trivial null hypotheses
Distinguish between biological (or other) significance and statistical significance
Always provide estimates of effect sizes and their precision, statistical significance (or lack thereof) notwithstanding
Consider using randomization and/or resampling methods to generate actual distribution of test statistics.
Bio 4118 Applied Biostatistics
2001