1. Hypothesis Testing: Cautions
STAT 250
Dr. Kari Lock Morgan
Hypothesis Testing: Cautions
SECTION 4.3, 4.5
Errors (4.3)
Multiple testing (4.5)
Replication

2. Intervals and Tests
Confidence intervals are most useful when you want to estimate population parameters
Hypothesis tests and p-values are most useful when you want to test hypotheses about population parameters
Confidence intervals give you a range of plausible values; p-values quantify the strength of evidence against the null hypothesis

3. Interval, Test, or Neither?
Is the following question best assessed using a confidence interval, a hypothesis test, or is statistical inference not relevant? How much do college students sleep, on average?
Confidence interval
Hypothesis test
Statistical inference not relevant

4. Interval, Test, or Neither?
Is the following question best assessed using a confidence interval, a hypothesis test, or is statistical inference not relevant? Do college students sleep more than the recommended 8 hours a night, on average?
Confidence interval
Hypothesis test
Statistical inference not relevant

5. Interval, Test, or Neither?
Is the following question best assessed using a confidence interval, a hypothesis test, or is statistical inference not relevant? What proportion of college students in the sleep study sample slept at least 8 hours?
Confidence interval
Hypothesis test
Statistical inference not relevant

6. Reproducibility Crisis

7. Reproducibility Crisis
Study: half of the studies you read about in the news are wrong (Vox, 3/3/2017)
Poor replication validity of biomedical association studies reported by newspapers (PLOS One, 2/21/2017)
The fickle p-value generates irreproducible results (Nature, 2/26/2015)
Why most published research findings are false (PLOS Medicine, 8/30/2005)

8. Does choice of mate improve offspring fitness (in fruit flies)?
Question of the Day
Does choice of mate improve offspring fitness (in fruit flies)?

9. Mate Choice and Offspring
What effect (if any) do you think freedom to choose a mate has on offspring fitness?
Improves it
Worsens it
Does not affect it

10. Original Study p-value < 0.01
Controversial – went against conventional wisdom
Researchers at Penn State tried to replicate the results…
Partridge, L. Mate choice increases a component of offspring fitness in fruit flies Nature, 283: /17/80.

12. Fruit Fly Mate Choice Experiment
Took 600 female fruit flies and randomly divided them into two groups:
300 got put in a cage with 900 males (mate choice)
300 were placed in individual vials with only one male each (no mate choice)
After mating, females were separated from the males and put in egg-laying chambers
200 larvae from each chamber was taken and placed in a cage with 200 mutant flies (for competition)
This was repeated 10 times/day for 5 days (50 runs)
Schaeffer, S.W., Brown, C.J., Anderson, W.W. (1984). “Does mate choice affect fitness?” Genetics, 107: s94. (Conducted at PSU by Dr. Steve Schaeffer in Biology)

13. Mate Choice and Offspring Survival
6,067 of the 10,000 mate choice larvae survived and 5,976 of the 10,000 no mate choice larvae survived
p-value: 0.102

14. Mate Choice and Offspring Survival
Another possibility: consider each run of the experiment a case, rather than each fly
Paired data, so look at difference for each pair
p-value = 0.21

15. Errors   Decision Truth
Errors can happen! There are four possibilities:
Decision
Reject H0
Do not reject H0
H0 true
H0 false 
TYPE I ERROR
Truth 
TYPE II ERROR
A Type I Error is rejecting a true null (false positive)
A Type II Error is not rejecting a false null (false negative)

16. Mate Choice and Offspring Fitness
Option #1: The original study (p-value < 0.01) made a Type I error, and H0 is really true
Option #2: The second study (p-value = or 0.21) made a Type II error, and Ha is really true
Option #3: No errors were made; different experimental settings yielded different results
Same species of fruit fly, same type of mutant, same design
Possible difference: The original study had flies that had been in the lab for longer, so were more likely to be at genetic equilibrium
[Note: Dr. Schaeffer suspects Option #1, saying the original study is an outlier among studies of this kind]

17. Analogy to Law A person is innocent until proven guilty.
Evidence must be beyond the shadow of a doubt.
Types of mistakes in a verdict?
Convict an innocent
Release a guilty

18. Probability of Type I Error
Distribution of statistics, assuming H0 true:
If the null hypothesis is true:
5% of statistics will be in the most extreme 5%
5% of statistics will give p-values less than 0.05
5% of statistics will lead to rejecting H0 at α = 0.05
If α = 0.05, there is a 5% chance of a Type I error

19. Probability of Type I Error
Distribution of statistics, assuming H0 true:
If the null hypothesis is true:
1% of statistics will be in the most extreme 1%
1% of statistics will give p-values less than 0.01
1% of statistics will lead to rejecting H0 at α = 0.01
If α = 0.01, there is a 1% chance of a Type I error

20. Probability of Type I Error
The probability of making a Type I error (rejecting a true null) is the significance level, α

21. Probability of Type II Error
How can we reduce the probability of making a Type II Error (not rejecting a false null)?
Decrease the sample size
Increase the sample size

22. Larger sample size makes it easier to reject the null
H0: p = 0.5
Ha: p > 0.5
n = 100
So, increase n to decrease chance of Type II error

23. Probability of Type II Error
How can we reduce the probability of making a Type II Error (not rejecting a false null)?
Decrease the significance level
Increase the significance level

24. Significance Level and Errorsα
Reject H0
Could be making a Type I error if H0 true
Chance of Type I error
Do not reject H0
Could be making a Type II error if Ha true
Related to chance of making a Type II error
Decrease α if Type I error is very bad
Increase α if Type II error is very bad

25. Multiple Testing Because the chance of a Type I error is α…
α of all tests with true null hypotheses will yield significant results just by chance.
If 100 tests are done with α = 0.05 and nothing is really going on, 5% of them will yield significant results, just by chance
This is known as the problem of multiple testing

26. Multiple Testing
Consider a topic that is being investigated by research teams all over the world
Using α = 0.05, 5% of teams are going to find something significant, even if the null hypothesis is true

27. Multiple Testing
Consider a research team/company doing many hypothesis tests
Using α = 0.05, 5% of tests are going to be significant, even if the null hypotheses are all true

28. Mate Choice and Offspring Fitness
The experiment was actually comprised of 50 smaller experiments. What if we had calculated the p-value for each run?
50 p-values:
What if we just reported the run that yielded a p-value of ?
Is that ethical?

29. Publication Bias
Publication bias refers to the fact that usually only the significant results get published
The one study that turns out significant gets published, and no one knows about all the insignificant results (also known as the file drawer problem)
This combined with the problem of multiple testing can yield very misleading results

30. Jelly Beans Cause Acne! http://xkcd.com/882/
Consider having your students act this out in class, each reading aloud a different part. it’s very fun!

31. Multiple Testing and Publication Bias
α of all tests with true null hypotheses will yield significant results just by chance.
The one that happens to be significant is the one that gets published.
THIS SHOULD SCARE YOU.

32. Clinical Trials Preclinical (animal studies)
Phase 0: Study pharmacodynamics and pharmacokinetics
Phase 1: Screening for safety
Phase 2: Placebo trials to establish efficacy
Phase 3: Trials against standard treatment and to confirm efficacy
Only then does a drug go to market…

33. What Can You Do? Point #1: Errors (type I and II) are possible
Point #2: Multiple testing and publication bias are a huge problem
Is it all hopeless? What can you do?
Recognize when a claim is one of many tests
Adjust for multiple tests (e.g. Bonferroni)
Look for replication of results…

34. Replication
Replication (or reproducibility) of a study in another setting or by another researcher is extremely important!
Studies that have been replicated with similar conclusions gain credibility
Studies that have been replicated with different conclusions lose credibility
Replication helps guard against Type I errors AND helps with generalizability

35. Mate Choice and Offspring Fitness
Actually, the research at Penn State included 3 different experiments; two different species of fruit flies and three different mutant types
1. Drosophila melanogaster, Mutant: sparkling eyes
2. Drosophila melanogaster, Mutant: white eyes
3. Drosophila pseudoobscura, Mutant: orange eyes
Multiple possible outcomes (% surviving in each group, % of survivors who were from experimental group (not mutants)
Multiple ways to analyze – proportions, quantitative paired analysis

36. Mate Choice and Offspring Fitness
Original study: Significant in favor of choice
p-value < 0.01
PSU study #1: Not significant
6067/ / = = 0.009
p-value = 0.09
PSU study #2: Significant in favor of no choice
4579/10000 – 4749/10000 = – =
p-value = for choice, for no choice
PSU study #3: Significant in favor of no choice
1641/5000 – 1758/5000 = – = -0.02
p-value = for choice, for no choice

37. Reproducibility Crisis
“While the public remains relatively unaware of the problem, it is now a truism in the scientific establishment that many preclinical biomedical studies, when subjected to additional scrutiny, turn out to be false. Many researchers believe that if scientists set out to reproduce preclinical work published over the past decade, a majority would fail. This, in short, is the reproducibility crisis."Amid a Sea of False Findings, the NIH Tries Reform (3/16/15)
A recent study tried to replicate 100 results published in psychology journals: 97% of the original results were significant, only 36% of replicated results were significant
Estimating the reproducibility of psychological science (8/28/15)

38. Summary Conclusions based off p-values are not perfect
Type I and Type II errors can happen
α of all tests will be significant just by chance
Often, only the significant results get published
Replication is important for credibility

39. To Do
HW 4.4, 4.5 (due Monday, 3/20)

40.
Author: JB Landers