1. Multiple testing adjustments European Molecular Biology Laboratory Predoc Bioinformatics Course 17 th Nov 2009 Tim Massingham, tim.massingham@ebi.ac.uk

2. Motivation Already come across several cases where need to correct p-values Exp 1Exp 2Exp 3Exp 4Exp 5Exp 6 Exp 10.0270.0330.4090.3300.784 Exp 20.1170.8410.9850.004 Exp 30.8690.9270.001 Exp 40.2450.021 Exp 50.004 Exp 6 Pairwise gene expression data What happens if we perform several vaccine trials?

3. Motivation 10 new vaccines are trialled Declare vaccine a success if test has p-value of less than 0.05 If none of the vaccines work, what is our chance of success?

4. Motivation 10 new vaccines are trialled Declare vaccine a success if test has p-value of less than 0.05 Each trial has probability of 0.05 of “success” (false positive) Each trial has probability of 0.95 of “failure” (true negative) Probability of at least one= 1 - Probability of none = 1 - (Probability a trial unsuccessful) 10 = 1 - 0.95 10 = 0.4 If none of the vaccines work, what is our chance of a “success”? Rule of Thumb Multiple size of test by number of tests

5. Motivation More extreme example: test entire population for disease True negativeFalse positive False negativeTrue positive Mixture: some of population have disease, some don’t Find individuals with disease Family Wise Error Rate Control probability that any false positive occurs False Discovery Rate Control proportion of false positives discovered True status Healthy Diseased Test report Healthy Diseased FDR = # false positives = # false positives # positives # true positives + # false positives

6. Cumulative distribution Simple examination by eye: The cumulative distribution should be approximately linear Rank Rank data Plot rank against p-value P-value 01 1 n N.B. Often scale ranks to (0,1] by dividing by largest rank Start(0,1) End(1,n) Never decreases

7. Cumulative distribution Five sets of uniformly distributed p-values Non-uniformly distributed data. Excess of extreme p-values (small) Examples: For 910 p-values Could use a one-sided Kolmogorov test if desired

8. A little set theory Test 1 false positive Test 2 false positive Test 3 false positive No test gives false positive All tests give false positive Represent all possible outcomes of three tests in a Venn diagram Areas are probabilities of events happening

9. A little set theory + + ≤ P(any test gives a false positive)

10. A little set theory + + ≤

11. Bonferroni adjustment Want to control thisKnow how to control each of these (the size of each test) Keep things simple: do all tests at same size If we have n tests, each at size nthen

12. Bonferroni adjustment If we have n tests, each at size nthen Family-Wise Error Rate

13. Example 1 Look at deviations from Chargaff’s 2 nd parity rule A and T content of genomes for 910 bugs Many show significant deviations First 9 pvalues 3.581291e-66 3.072432e-12 1.675474e-01 6.687600e-01 1.272040e-05 1.493775e-23 2.009552e-26 1.024890e-14 1.519195e-24 Unadjusted pvalues pvalue < 0.05764 pvalue < 0.01717 pvalue < 1e-5559 Bonferroni adjusted pvalues pvalue < 0.05582 pvalue < 0.01560 pvalue < 1e-5461 First 9 adjusted pvalues 3.258975e-63 2.795913e-09 1.000000e+00 1.000000e+00 1.157556e-02 1.359335e-20 1.828692e- 23 9.326496e-12 1.382467e-21

14. Aside: pvalues measure evidence Shown that many bugs deviate substantial from Chargaff’s 2 nd rule p-values tell us that there is significant evidence for a deviation median Upper quantile Lower quantile Lots of bases and so ability to detect small deviations from 50% Powerful test 1st Qu. Median 3rd Qu. 0.4989 0.4999 0.5012

15. Bonferroni is conservative Conservative: actual size of test is less than bound Not too bad for independent tests Worst when positively correlated Applying same test to subsets of data Applying similar tests to same data More subtle problem Mixture of blue and red circles Null hypothesis: Is blue Red circles are never false positives

16. Bonferroni is conservative + + ≤ If experiment really is different from null, then Over adjusted p-value Number of potential false positives may be less than number of tests

17. Holm’s method Holm(1979) suggests repeatedly applying Bonferroni Initial Bonferroni:InsignificantSignificant InsignificantSignificant No false positive?Been overly strict, apply Bonferroni only to insignificant set. False positive?More won’t hurt, so may as well test again Step 2 InsignificantSignificantStep 3 Stop when “insignificant” set does not shrink further

18. Example 2 Bonferroni adjusted pvalues pvalue < 0.05582 pvalue < 0.01560 pvalue < 1e-5461 First 9 adjusted pvalues 3.258975e-63 2.795913e-09 1.000000e+00 1.000000e+00 1.157556e-02 1.359335e-20 1.828692e- 23 9.326496e-12 1.382467e-21 Return to Chargaff data 910 bugs but more than half are significantly different after adjustment There is strong evidence that we’ve over-corrected First 9 Holm adjusted pvalues 2.915171e-63 1.591520e-09 1.000000e+00 1.000000e+00 4.452139e-03 9.903730e-21 1.390610e- 23 5.623765e-12 1.019380e-21 Holm adjusted pvalues pvalue < 0.05606 (+24) pvalue < 0.01574 (+14) pvalue < 1e-5472 (+12) Gained a couple of percent more but notice that gains tail off

19. Hochberg’s method Consider a pathological case Apply same test to same data multiple times # Ten identical pvalues pvalues <- rep(0.01,10) # None are significant with Bonferroni p.adjust(pvalues,method=“bonferroni”) 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 # None are significant with Holm p.adjust(pvalues,method=“holm”) 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 # Hochberg recovers correctly adjusted pvalues p.adjust(pvalues,method=“hochberg”) 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01 First 9 Hochberg adjusted pvalues 2.915171e-63 1.591520e-09 9.972469e-01 9.972469e- 01 4.452139e-03 9.903730e-21 1.390610e-23 5.623765e-12 1.019380e-21 Hochberg adjusted pvalues pvalue < 0.05606 pvalue < 0.01574 pvalue < 1e-5472 Hochberg adjustment is identical to Holm for Chargaff data …. but requires additional assumptions

20. False Discovery Rates New methods, dating back to 1995 Gaining popularity in literature but mainly used for large data sets Useful for enriching data sets for further analysis Recap FWER:control probability of any false positive occurring FDR:control proportion of false positives that occur “q-value” is proportion of significant tests expected to be false positives q-value times number significant = expected number of false positives Methods Benjamini & Hochberg (1995) Benjamini & Yekutieli (2001) Storey (2002,2003) aka “positive false discovery rate”

21. Example 3 Returning once more to the Chargaff data First 9 FDR q-values 3.359768e-65 7.114283e-12 1.891664e-01 6.931340e- 01 2.063380e-05 5.481191e-23 8.350193e-26 2.569283e-14 5.760281e-24 FDR q-values qvalue < 0.05759 qvalue < 0.01713 qvalue < 1e-5547 Q-values have a different interpretation from p-values Use qvalues to get the expected number of false positives qvalue = 0.05expect 38 false positives (759 x 0.05) qvalue = 0.01expect 7 false positives (713 x 0.01) qvalue = 1e-5expect 1/200 false positives

22. Summary Holm is always better than Bonferroni Hochberg can be better but has additional assumptions FDR is a more powerful approach - finds more things significant controls a different criteria more useful for exploratory analyses than publications A little question Suppose results are published if the p-value is less than 0.01, what proportion of the scientific literature is wrong?