1. The False Discovery Rate A New Approach to the Multiple Comparisons Problem Thomas Nichols Department of Biostatistics University of Michigan

2. fMRI Multiple Comparisons Problem 4-Dimensional Data –1,000 multivariate observations, each with 100,000 elements –100,000 time series, each with 1,000 observations Massively Univariate Approach –100,000 hypothesis tests Massive MCP! 4-Dimensional Data –1,000 multivariate observations, each with 100,000 elements –100,000 time series, each with 1,000 observations Massively Univariate Approach –100,000 hypothesis tests Massive MCP! 1,000 1 2 3...

3. Solutions for Multiple Comparison Problem A MCP Solution Must Control False Positives –How to measure multiple false positives? Familywise Error Rate (FWER) –Chance of any false positives –Controlled by Bonferroni & Random Field Methods False Discovery Rate (FDR) –Proportion of false positives among rejected tests A MCP Solution Must Control False Positives –How to measure multiple false positives? Familywise Error Rate (FWER) –Chance of any false positives –Controlled by Bonferroni & Random Field Methods False Discovery Rate (FDR) –Proportion of false positives among rejected tests

4. False Discovery Rate Illustration: Signal+Noise Noise

5. FWE 6.7% 10.4%14.9%9.3%16.2%13.8%14.0% 10.5%12.2%8.7% Control of Familywise Error Rate at 10% 11.3% 12.5%10.8%11.5%10.0%10.7%11.2%10.2%9.5% Control of Per Comparison Rate at 10% Percentage of Null Pixels that are False Positives Control of False Discovery Rate at 10% Occurrence of Familywise Error Percentage of Activated Pixels that are False Positives

6. Benjamini & Hochberg Procedure Select desired limit q on E(FDR) Order p-values, p (1)  p (2) ...  p (V) Let r be largest i such that Reject all hypotheses corresponding to p (1),..., p (r). Select desired limit q on E(FDR) Order p-values, p (1)  p (2) ...  p (V) Let r be largest i such that Reject all hypotheses corresponding to p (1),..., p (r). p (i)  i/V  q/c(V) p(i)p(i) i/Vi/V i/V  q/c(V) p-value 01 0 1 JRSS-B (1995) 57:289-300

7. Benjamini & Hochberg Procedure c(V) = 1 –Positive Regression Dependency on Subsets Technical condition, special cases include –Independent data –Multivariate Normal with all positive correlations Result by Benjamini & Yekutieli. c(V) =  i=1,...,V 1/i  log(V)+0.5772 –Arbitrary covariance structure c(V) = 1 –Positive Regression Dependency on Subsets Technical condition, special cases include –Independent data –Multivariate Normal with all positive correlations Result by Benjamini & Yekutieli. c(V) =  i=1,...,V 1/i  log(V)+0.5772 –Arbitrary covariance structure

8. Benjamini & Hochberg: Varying Signal Extent Signal Intensity3.0Signal Extent 1.0Noise Smoothness3.0 p = z = 1

9. Benjamini & Hochberg: Varying Signal Extent Signal Intensity3.0Signal Extent 2.0Noise Smoothness3.0 p = z = 2

10. Benjamini & Hochberg: Varying Signal Extent Signal Intensity3.0Signal Extent 3.0Noise Smoothness3.0 p = z = 3

11. Benjamini & Hochberg: Varying Signal Extent Signal Intensity3.0Signal Extent 5.0Noise Smoothness3.0 p = 0.000252z = 3.48 4

12. Benjamini & Hochberg: Varying Signal Extent Signal Intensity3.0Signal Extent 9.5Noise Smoothness3.0 p = 0.001628z = 2.94 5

13. Benjamini & Hochberg: Varying Signal Extent Signal Intensity3.0Signal Extent16.5Noise Smoothness3.0 p = 0.007157z = 2.45 6

14. Benjamini & Hochberg: Varying Signal Extent Signal Intensity3.0Signal Extent25.0Noise Smoothness3.0 p = 0.019274z = 2.07 7

15. Benjamini & Hochberg: Properties Adaptive –Larger the signal, the lower the threshold –Larger the signal, the more false positives False positives constant as fraction of rejected tests Not a problem with imaging’s sparse signals Smoothness OK –Smoothing introduces positive correlations Adaptive –Larger the signal, the lower the threshold –Larger the signal, the more false positives False positives constant as fraction of rejected tests Not a problem with imaging’s sparse signals Smoothness OK –Smoothing introduces positive correlations

16. FDR: Example Verbal fluency data 14 42-second blocks ABABAB... A: Two syllable words presented aurally B: Silence Imaging parameters –2Tesla scanner, TR = 7 sec –84 64x64x64 images of 3 x 3 x 3 mm voxels Verbal fluency data 14 42-second blocks ABABAB... A: Two syllable words presented aurally B: Silence Imaging parameters –2Tesla scanner, TR = 7 sec –84 64x64x64 images of 3 x 3 x 3 mm voxels

17. FDR Example: Plot of FDR Inequality p (i)  ( i/V ) ( q/c(V) )

18. FDR: Example FDR  0.05 Indep/PRDS t 0 = 3.8119 FWER  0.05 Bonferroni t 0 = 5.485 FDR  0.05 Arbitrary Cov. t 0 = 5.0747

19. FDR Software for SPM http://www.sph.umich.edu/~nichols/FDR

20. FDR: Conclusions False Discovery Rate –A new false positive metric Benjamini & Hochberg FDR Method –Straightforward solution to fNI MCP –Just one way of controlling FDR New methods under development e.g. C. Genovese or J. Storey Limitations –Arbitrary dependence result less sensitive False Discovery Rate –A new false positive metric Benjamini & Hochberg FDR Method –Straightforward solution to fNI MCP –Just one way of controlling FDR New methods under development e.g. C. Genovese or J. Storey Limitations –Arbitrary dependence result less sensitive http://www.sph.umich.edu/~nichols/FDR Prop Ill Start

21. References Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57:289--300. Benjamini, Y, Yekutieli D (2002). The control of the false discovery rate under dependence. Annals of Statistics. To appear. Genovese CR, Lazar N, Nichols TE (2002). Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate. NeuroImage, 15:870-878. Benjamini Y, Hochberg Y (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57:289--300. Benjamini, Y, Yekutieli D (2002). The control of the false discovery rate under dependence. Annals of Statistics. To appear. Genovese CR, Lazar N, Nichols TE (2002). Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate. NeuroImage, 15:870-878.

22. Positive Regression Dependency Does fMRI data exhibit total positive correlation? Example –160 scan experiment –Spatial autocorrelation of residuals –Single voxel with all others Negative correlation exists! Does fMRI data exhibit total positive correlation? Example –160 scan experiment –Spatial autocorrelation of residuals –Single voxel with all others Negative correlation exists!