1. Hypothesis Testing: p-value
2/13/12
Randomization distribution
p-value
Statistical significance
Section 4.2
Professor Kari Lock Morgan
Duke University

2. To Do Project 1 Proposal (due Wednesday) Homework 4 (due Monday)
NO LATE HOMEWORK ACCEPTED!

3. Exercise and Pulse
In the actual experiment, the people who exercised for 5 seconds had an average pulse of Those who did not exercise had an average pulse of 69.6.
Is this sample difference larger than we would see, just by random chance, if exercising for 5 seconds did not increase pulse rate?

4. Exercise and Pulse p-value
If 5 seconds of exercise does not increase pulse rate, we would see a sample difference as extreme as 15.9 in only of all such experiments.

5. Cocaine Addiction
In a randomized experiment on treating cocaine addiction, 48 people were randomly assigned to take either Desipramine (a new drug), or Lithium (an existing drug), and then followed to see who relapsed
Question of interest: Is Desipramine better than Lithium at treating cocaine addiction?

6. Cocaine Addiction What is the statistic of interest?
What are the hypotheses of this test?

7. R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R
1. Randomly assign units to treatment groups
Desipramine
Lithium R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R

8. 3. Observe relapse counts in each group
2. Conduct experiment
3. Observe relapse counts in each group
R = Relapse
N = No Relapse
1. Randomly assign units to treatment groups
Desipramine
Lithium R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R N R N R R R R R R R R R R R R R N N R N N N N N N N N R R R R R R R R R R R R N N N N N N N N N N N N N N N N N N N N N N N N
10 relapse, 14 no relapse
18 relapse, 6 no relapse

9. Cocaine Addiction Two options:
H0 is true (the drugs cause the same proportion of relapses)
Ha is true (Desipramine causes a smaller proportion of relapses than Lithium)
In situation (1), how would you explain the observed difference in the proportion of relapses?
How can we see whether this is a plausible explanation?

10. Measuring Evidence against H0
To see if a statistic provides evidence against H0, we need to see what kind of sample statistics we would observe, just by random chance,
if H0 were true

11. Randomization Test Assume the null hypothesis is true
Simulate new randomizations
For each, calculate the statistic of interest
Find the proportion of these simulated statistics that are as extreme as your observed statistic

12. R R R R R R R R R R R R R R R R N N R R R R R R N N N N N N R R R R R R N N N N N N N N N N N N
10 relapse, 14 no relapse
18 relapse, 6 no relapse

13. R R R R R R R R R R R R R R R R N N R R R R R R N N N N N N R R R R R R N N N N N N N N N N N N
Simulate another randomization
Desipramine
Lithium R N R N N N N R R R R R R R N R R N N N R N R R R N N R N R R N R N N N R R R N R R R R
16 relapse, 8 no relapse
12 relapse, 12 no relapse

14. Simulate another randomization
Desipramine
Lithium R R R R R R R R R R R R R N R R N N R R R R R R R R N R N R R R R R R R R N R N R R N N N N N N
17 relapse, 7 no relapse
11 relapse, 13 no relapse

15. Proportion as extreme as observed statistic
Cocaine Addiction
Proportion as extreme as observed statistic
p-value
observed statistic
The probability of getting a sample difference in proportions as low as just by random chance, if the drugs really are equally effective, is 0.02

16. p-value
Based on a randomization distribution, the p-value is the proportion of statistics that are more extreme than that observed
This is the area in the tail(s) beyond the observed statistic in the randomization distribution
Which tail(s) to include depends on the alternative hypothesis

17. Alternative Hypothesis
A one-sided alternative contains either > or <
A two-sided alternative contains ≠
The alternative hypothesis depends on the research question of interest
For a one-sided alternative, the p-value is the proportion in the tail specified by Ha
For a two-sided alternative, the p-value is twice the proportion in the smallest tail

18. Sleep or Caffeine for Memory?
Students were given words to memorize, and then randomly assigned to either take a 90 min nap, or to stay awake and take a caffeine pill. 2 ½ hours later, all students were tested on their recall ability. Is sleep or caffeine better for memory?
How extreme would this be if H0 were true???
Mednick, Cai, Kanady, and Drummond (2008). “Comparing the benefits of caffeine, naps and placebo on verbal, motor and perceptual memory,” Behavioral Brain Research, 193,

19. Sleep or Caffeine for Memory?
p-value
= 2 × 0.022
= 0.044
If sleep and caffeine are equally effective for memory, we would get a sample difference in means as extreme as 3 in about of all experiments.

20. Infections in Childbirth
The Center for Disease Control (CDC) conducted a randomized trial in South Africa in which half of women in labor are randomly assigned to be treated with a wipe containing chlorohexidine, and the other half with a sterile wipe (control)
Source: Eriksen, Sweeten, Blanco (1997). “Chlorohexidine vs Sterile Vaginal Wash During Labor to Prevent Peripartum Infection,” American Journal of Obstetrics and Gynecology, 176:

21. Infections in Childbirth
What can you conclude about the p-value?
p-value < 0.5
p-value > 0.5
Nothing

22. Alternative Hypothesis
The p-value is the probability of getting results are extreme as your sample statistic, if the null hypothesis is true
“As extreme as” is defined in the direction of the alternative hypothesis. (for two-sided alternatives, consider both tails)
If your sample statistic does not support your alternative hypothesis, there is no point in going through the test!

23. p-value What can you conclude about the p-value? p-value = 0
Nothing

24. p-value What can you conclude about the p-value? p-value < 0.5
Nothing

25. Multiple Sclerosis and Sunlight
It is believed that sunlight offers some protection against multiple sclerosis, but the reason is unknown
To find out, researchers randomly assigned mice to one of three treatments:
Control (nothing)
Vitamin D Supplements
UV Light
All mice were injected with proteins known to induce a mouse form of MS, and they observed which mice got MS
Seppa, Nathan. “Sunlight may cut MS risk by itself”, Science News, April 24, 2010 pg 9, reporting on a study appearing March 22, 2010 in the Proceedings of the National Academy of Science.

26. Multiple Sclerosis and Sunlight
In testing whether UV light provides protection against MS in mice, what are the null and alternative hypotheses?
pUV = proportion of mice exposed to UV light that get MS
pC = proportion of mice not exposed to UV light that get MS
H0 : pUV – pC > 0, Ha: pUV – pC = 0
H0 : pUV – pC < 0, Ha: pUV – pC = 0
H0 : pUV – pC = 0, Ha: pUV – pC > 0
H0 : pUV – pC = 0, Ha: pUV – pC < 0

27. Multiple Sclerosis and Sunlight
In testing whether UV light provides protection against MS in mice, the experiment yielded a p-value of What would you conclude?
H0 is probably not true  UV light does provide protection against MS
H0 is probably not true  UV light does not provide protection against MS
Ha is probably not true  UV light does provide protection against MS
Ha is probably not true  UV light does not provide protection against MS
Nothing

28. Multiple Sclerosis and Sunlight
In testing whether Vitamin D provides protection against MS in mice, the experiment yielded a p-value of What would you conclude?
H0 is probably not true  Vitamin D does provide protection against MS
H0 is probably not true  Vitamin D does not provide protection against MS
Ha is probably not true  Vitamin D does provide protection against MS
Ha is probably not true  Vitamin D does not provide protection against MS
Nothing

29. Strength of Evidence
The p-value is the probability of getting results as extreme as those observed, if the null hypothesis is true
The p-value measures evidence against the null hypothesis
p-value

30. The smaller the p-value, the stronger the evidence against Ho.

31. How small is small enough?
Hypothesis Testing
If the p-value is small enough, we reject the null hypothesis, in favor of the alternative hypothesis
How small is small enough?

32. Statistical Significance
The significance level, , is the threshold below which the p-value is deemed small enough to reject the null hypothesis
If the p-value is less than , the results are statistically significant, and we reject the null hypothesis in favor of the alternative
mention t-shirt

33. Statistical Significance

34. Formal Decisions For a given significance level, ,
p-value <   Reject Ho
p-value >   Do not Reject Ho

35. Statistical Conclusions
Strength of evidence against H0:
Formal decision of hypothesis test, based on  = 0.05 :

36. Formal Decisions
A formal hypothesis test has only two possible conclusions:
The p-value is small: reject the null hypothesis in favor of the alternative
The p-value is not small: do not reject the null hypothesis

37. Elephant Example Example: H0 : X is an elephant
Ha : X is not an elephant
Would you conclude, if you get the following data?
X has four legs
X walks on two legs

38. Summary
A randomization distribution shows the distribution of statistics that would be observed if H0 were true
A p-value is the probability of getting a statistic as extreme as that observed, if H0 is true
The p-value measures the strength of evidence against the null hypothesis
Results are statistically significant if the p-value is less than the significance level, α
In making formal decisions, reject H0 if the p-value is less than α, otherwise do not reject H0