1. Multiple comparisons in M/EEG analysis Gareth Barnes Wellcome Trust Centre for Neuroimaging University College London SPM M/EEG Course London, May 2013

2. Format  What problem ?- multiple comparisons and post-hoc testing.  Some possible solutions  Random field theory

3. Pre- processing General Linear Model Statistical Inference Contrast c Random Field Theory

4. T test on a single voxel / time point Task A Task B T value 1.66 Test only this Ignore these Null distribution of test statistic T u Threshold T distribution P=0.05

5. T test on many voxels / time points Task A Task B T value Test all of this Null distribution of test statistic T u Threshold ? T distribution P= ?

6. What not to do..

7. Don’t do this : With no prior hypothesis. Test whole volume. Identify SPM peak. Then make a test assuming a single voxel/ time of interest. James Kilner. Clinical Neurophysiology 2013. SPM t

9. What to do:

10. Multiple tests in space t  uu t  uu t  uu t  uu t  uu t  uu If we have 100,000 voxels, α =0.05  5,000 false positive voxels. This is clearly undesirable; to correct for this we can define a null hypothesis for a collection of tests. signal

11. P<0.05 (independent samples) signal Multiple tests in time

12. Bonferroni correction The Family-Wise Error rate (FWER), α FWE, for a family of N tests follows the inequality: where α is the test-wise error rate. Therefore, to ensure a particular FWER choose: This correction does not require the tests to be independent but becomes very stringent if they are not.

13. Family-Wise Null Hypothesis False positive Use of ‘corrected’ p-value, α =0.1 Use of ‘uncorrected’ p-value, α =0.1 Family-Wise Null Hypothesis: Activation is zero everywhere Family-Wise Null Hypothesis: Activation is zero everywhere If we reject a voxel null hypothesis at any voxel, we reject the family-wise Null hypothesis A FP anywhere in the image gives a Family Wise Error (FWE) Family-Wise Error rate (FWER) = ‘corrected’ p-value

14. P<0.05 200 (independent samples) Multiple tests- FWE corrected

15. What about data with different topologies and smoothness.. Smooth time Different smoothness In time and space Volumetric ROIs Surfaces Bonferroni works fine for independent tests but..

16. Non-parametric inference: permutation tests 5%  Parametric methods – Assume distribution of max statistic under null hypothesis  Nonparametric methods – Use data to find distribution of max statistic under null hypothesis – any max statistic to control FWER

17. Random Field Theory Keith Worsley, Karl Friston, Jonathan Taylor, Robert Adler and colleagues Number peaks= intrinsic volume * peak density In a nutshell :

18. Number peaks= intrinsic volume * peak density Currant bun analogy Number of currants = volume of bun * currant density

19. Number peaks= intrinsic volume* peak density How do we specify the number of peaks

20. EC=2 EC=1 Euler characteristic (EC) at threshold u = Number blobs- Number holes At high thresholds there are no holes so EC= number blobs How do we specify the number of peaks

21. Number peaks= intrinsic volume * peak density How do we specify peak (or EC) density

22. Number peaks= intrinsic volume * peak density The EC density - depends only on the type of random field (t, F, Chi etc ) and the dimension of the test. See J.E. Taylor, K.J. Worsley. J. Am. Stat. Assoc., 102 (2007), pp. 913–928 Number of currants = bun volume * currant density EC density, ρ d (u) tF X2X2 peak densities for the different fields are known

23. Number peaks= intrinsic volume * peak density Currant bun analogy Number of currants = bun volume * currant density

24. Which field has highest intrinsic volume ? AB CD More samples (higher volume) Smoother ( lower volume) Equal volume

25. LKC or resel estimates normalize volume The intrinsic volume (or the number of resels or the Lipschitz- Killing Curvature) of the two fields is identical

26. From Expected Euler Characteristic Intrinsic volume (depends on shape and smoothness of space) Depends only on type of test and dimension =2.9 (in this example) Threshold u Gaussian Kinematic formula

27. -- EC observed o EC predicted Expected Euler Characteristic Gaussian Kinematic formula Intrinsic volume (depends on shape and smoothness of space) EC=1-0 Depends only on type of test and dimension Number peaks= intrinsic volume * peak density From

28. Expected Euler Characteristic Gaussian Kinematic formula Intrinsic volume (depends on shape and smoothness of space) EC=2-1 Depends only on type of test and dimension 0.05 FWE threshold Expected EC Threshold i.e. we want one false positive every 20 realisations (FWE=0.05)

29. Getting FWE threshold From Know intrinsic volume (10 resels) 0.05 Want only a 1 in 20 Chance of a false positive Know test (t) and dimension (1) so can get threshold u (u)

30. Can get correct FWE for any of these.. mm time mm time frequency fMRI, VBM, M/EEG source reconstruction M/EEG 2D time-frequency M/EEG 2D+t scalp-time M/EEG 1D channel-time

31. Peace of mind Guitart-Masip M et al. J. Neurosci. 2013;33:442-451 P<0.05 FWE

32. Conclusions  Strong prior hypotheses can reduce the multiple comparisons problem.  Random field theory is a way of predicting the number of peaks you would expect in a purely random field (without signal).  Can control FWE rate for a space of any dimension and shape.

33. Acknowledgments Guillaume Flandin Vladimir Litvak James Kilner Stefan Kiebel Rik Henson Karl Friston Methods group at the FIL

34. References  Friston KJ, Frith CD, Liddle PF, Frackowiak RS. Comparing functional (PET) images: the assessment of significant change. J Cereb Blood Flow Metab. 1991 Jul;11(4):690-9.  Worsley KJ, Marrett S, Neelin P, Vandal AC, Friston KJ, Evans AC. A unified statistical approach for determining significant signals in images of cerebral activation. Human Brain Mapping 1996;4:58-73.  Chumbley J, Worsley KJ, Flandin G, and Friston KJ. Topological FDR for neuroimaging. NeuroImage, 49(4):3057-3064, 2010.  Chumbley J and Friston KJ. False Discovery Rate Revisited: FDR and Topological Inference Using Gaussian Random Fields. NeuroImage, 2008.  Kilner J and Friston KJ. Topological inference for EEG and MEG data. Annals of Applied Statistics, in press.  Bias in a common EEG and MEG statistical analysis and how to avoid it. Kilner J. Clinical Neurophysiology 2013.

35. Peak, cluster and set level inference Peak level test: height of local maxima Cluster level test: spatial extent above u Set level test: number of clusters above u Sensitivity  Regional specificity  : significant at the set level : significant at the cluster level : significant at the peak level L 1 > spatial extent threshold L 2 < spatial extent threshold peak set EC threshold

36. Random Field Theory  The statistic image is assumed to be a good lattice representation of an underlying continuous stationary random field. Typically, FWHM > 3 voxels  Smoothness of the data is unknown and estimated: very precise estimate by pooling over voxels  stationarity assumptions (esp. relevant for cluster size results).  But can also use non-stationary random field theory to correct over volumes with inhomogeneous smoothness  A priori hypothesis about where an activation should be, reduce search volume  Small Volume Correction: