Randomization Tests Dr. Kari Lock Morgan PSU 016 11/5/14

Extrasensory Perception Is there such a thing as extrasensory perception (ESP) or a “sixth sense”? Do you believe in ESP?

Extrasensory Perception One way to test for ESP is with Zener cards: Subjects draw a card at random and telepathically communicate this to someone who then guesses the symbol

Extrasensory Perception Let’s do our own study! Make your own Zener cards: Randomly choose a symbol Find a partner, telepathically communicate your symbol (no auditory or visual clues!), and have them guess your symbol. Switch roles. Did you guess correctly?

Extrasensory Perception There are five cards with five different symbols If there is no such thing as ESP, what proportion of guesses should be correct? Because there are 5 cards, each person has a 1/5 chance of guessing correctly each time, if ESP does not exist. H0: p = 1/5 Ha: p > 1/5

Extrasensory Perception Statistics vary from sample to sample: even if the population proportion is 1/5, not every sample proportion will be exactly 1/5 How do we determine when a sample proportion is far enough above 1/5 to provide evidence of ESP? More general: How do we determine when a sample statistic is far enough away from H0 to be statistically significant?

SIMULATE what would happen if H0 were true! Key Question How unusual is it to see a sample statistic as extreme as that observed, if H0 is true? How do we know how unusual a sample statistic would be if H0 were true? SIMULATE what would happen if H0 were true!

ESP: Simulate! How could we simulate what would happen, just by random chance, if the null hypotheses were true for the ESP experiment? Randomly choose a symbol. Return it to the rest, shuffle, and choose again for the (random) guess. Did you (randomly) get the correct symbol?

www.lock5stat.com/statkey Lots of simulations! We need many more simulations! www.lock5stat.com/statkey

ESP – Random Chance Are our results statistically significant? What can we conclude?

Randomization Distribution A randomization distribution is a collection of statistics from samples simulated assuming the null hypothesis is true

p-value The p-value is the chance of obtaining a sample statistic as extreme as (or more extreme than) the observed sample statistic, if the null hypothesis is true

Calculating a p-value What kinds of statistics would we get, just by random chance, if the null hypothesis were true? (randomization distribution) What proportion of these statistics are as extreme as our original sample statistic? (p-value)

Proportion as extreme as observed statistic ESP p-value Distribution of statistics that would be observed, just by random chance, if H0 true p-value = 0.247 Proportion as extreme as observed statistic p-value If you were all just guessing randomly, the chance of us getting a sample proportion as high as 0.294 is 0.247. observed statistic

Cocaine Addiction In a randomized experiment on treating cocaine addiction, 48 people were randomly assigned to take either Desipramine (a new drug), or Lithium (an existing drug), and then followed to see who relapsed Is Desipramine better than Lithium at treating cocaine addiction? pD, pL: proportion of cocaine 𝑝 𝐷 addicts who relapse after taking Desipramine or Lithium, respectively H0: pD = pL Ha: pD < pL

R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R 1. Randomly assign units to treatment groups Desipramine Lithium R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R

3. Observe relapse counts in each group 2. Conduct experiment 3. Observe relapse counts in each group R = Relapse N = No Relapse 1. Randomly assign units to treatment groups Desipramine Lithium R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R R N R N R R R R R R R R R R R R R N N R N N N N N N N N R R R R R R R R R R R R N N N N N N N N N N N N N N N N N N N N N N N N 10 relapse, 14 no relapse 18 relapse, 6 no relapse

Measuring Evidence against H0 To see if a statistic provides evidence against H0, we need to see what kind of sample statistics we would observe, just by random chance, if H0 were true

Cocaine Addiction “by random chance” means by the random assignment to the two treatment groups “if H0 were true” means if the two drugs were equally effective at preventing relapses (equivalently: whether a person relapses or not does not depend on which drug is taken) Simulate what would happen just by random chance, if H0 were true…

R R R R R R R R R R R R R R R R N N R R R R R R N N N N N N R R R R R R N N N N N N N N N N N N 10 relapse, 14 no relapse 18 relapse, 6 no relapse

R R R R R R R R R R R R R R R R N N R R R R R R N N N N N N R R R R R R N N N N N N N N N N N N Simulate another randomization Desipramine Lithium R N R N N N N R R R R R R R N R R N N N R N R R R N N R N R R N R N N N R R R N R R R R 16 relapse, 8 no relapse 12 relapse, 12 no relapse

Simulate another randomization Desipramine Lithium R R R R R R R R R R R R R N R R N N R R R R R R R R N R N R R R R R R R R N R N R R N N N N N N 17 relapse, 7 no relapse 11 relapse, 13 no relapse

Cocaine Addiction Why did you re-deal your cards? Shuffle your cards and deal them into two piles. What is your sample difference in proportions? Why did you re-deal your cards? Why did you leave the outcomes (relapse or no relapse) unchanged on each card? You want to know what would happen by random chance if the null hypothesis is true

www.lock5stat.com/statkey Lots of simulations! We need many more simulations! www.lock5stat.com/statkey

Proportion as extreme as observed statistic www.lock5stat.com/statkey Distribution of statistics that would be observed, just by random chance, if H0 true Proportion as extreme as observed statistic p-value observed statistic If the two drugs are equal regarding cocaine relapse rates, we have a 1.3% chance of seeing a difference in proportions as extreme as that observed.

Randomization Test p-values can be calculated by randomization distributions: Create a randomization distribution by simulating statistics you would see, just by random chance, if H0 were true Find the p-value as the proportion of simulated statistics as extreme as the observed statistic This idea works for any parameter!

Your Turn! Correlation NFL Teams Do NFL teams with more malevolent uniforms get more penalty yards? r = 0.43

p-value and Ha H0:  = 0 Upper-tail Ha:  > 0 (Right Tail) 𝑥 =2 𝑥 =−1 Lower-tail (Left Tail) H0:  = 0 Ha:  ≠ 0 𝑥 =2 Two-tailed

Summary p-values can be calculated by randomization distributions: Create a randomization distribution by simulating statistics you would see, just by random chance, if H0 were true Find the p-value as the proportion of simulated statistics as extreme as the observed statistic