Presentation is loading. Please wait.

Presentation is loading. Please wait.

Introduction to Permutation for Neuroimaging HST

Published byAngela Kathleen Baldwin Modified over 6 years ago

Similar presentations

Presentation on theme: "Introduction to Permutation for Neuroimaging HST"— Presentation transcript:

1 Introduction to Permutation for Neuroimaging HST583 2017
Douglas N. Greve Martinos Center

2 Outline Two examples Implementation Why permutation works Limitations

3 fMRI Analysis Overview
Subject 1 Preprocessing MC, STC, B0 Smoothing Normalization First Level GLM Analysis Raw Data C X Subject 2 Preprocessing MC, STC, B0 Smoothing Normalization First Level GLM Analysis Raw Data C X Higher Level GLM Subject 3 C X Preprocessing MC, STC, B0 Smoothing Normalization First Level GLM Analysis Correction for Multiple Comparisons Raw Data C X Subject 4 Preprocessing MC, STC, B0 Smoothing Normalization First Level GLM Analysis Cluster p<.05? Raw Data C X

4 fMRI Analysis Overview
Subject 1 Preprocessing MC, STC, B0 Smoothing Normalization First Level GLM Analysis Raw Data C X Subject 2 Preprocessing MC, STC, B0 Smoothing Normalization First Level GLM Analysis Raw Data C X Higher Level GLM Subject 3 C X Preprocessing MC, STC, B0 Smoothing Normalization First Level GLM Analysis Correction for Multiple Comparisons Raw Data C X Subject 4 Preprocessing MC, STC, B0 Smoothing Normalization First Level GLM Analysis Cluster p<.05? Raw Data C X

5 Two examples of parametric statistics and their permutation counterparts

6 Parametric: GLM and t-Test
Convert t to a p-value using Student’s t-Test

7 Parametric GLM: Two Group GLM Analysis = y = X*b
Does Group 1 differ from Group 2? C = [1 -1], Contrast = C*b = bG1- bG2 Compute T from t-test t-test assumes: Gaussian (skew=0), independent, homoscedastic If not, then p-values are not accurate = 1 bG1 bG2 y = X*b

8 Two Group GLM using Permutation
= 1 bG1 bG2 Shuffle y (permutation) Run analysis Compute simulation test statistic Ts Go back to step 1 Repeat a large (~10k) times, get 10k values of Ts Analyze your true data Compute observed test statistic To p value = How often To >Ts Justification: under the NULL, labelings in design matrix are irrelevant

9 Cluster Correction for Multiple Comparisons
Sig Map pVox < .001 Cluster Map

10 Cluster Table Cluster Size (mm3) 1 51368 2 41184 3 3784 4 1768 R L

11 Parametric: Gaussian Random Field Theory
pcluster = f(aVox,N,FWHM,ClusterSize) aVox Voxel-wise, Cluster-forming Threshold N – Search space. FWHM – Smoothing level ClusterSize – size of cluster to be tested pcluster – Cluster p-value Assumptions: Voxel-wise threshold sufficiently stringent (eg, .001, not .01) Smoothing level sufficiently high (eg, 2 voxels) Spatial smoothness is Gaussian (not to be confused with Gaussian noise assumptions)

12 Cluster Table Cluster Size (mm3) p-value 1 51368 .00015 2 41184 .00250
3784 .02600 4 1768 .04900 R L

13 Eklund Significance Statement
“Here, we used resting-state fMRI data from 499 healthy controls to conduct 3 million task group analyses. Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results.” Cluster Failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Eklund, Nichols, Knutsson, PNAS Problem extends to at least thickness and VBM analysis

14 Cluster Analysis using Permutation
= 1 bG1 bG2 Shuffle y (permutation) Run analysis at every voxel Threshold, find max cluster size Ts Go back to step 1 Repeat a large (~10k) times, get 10k values of Ts Analyze your true data Compute observed cluster size To Cluster p value = How often To >Ts Permutation fixed problems found by Eklund, et al, 2016 as well as in thickness.

15 How permutation is implemented

16 … First, what is the “best” way to compute p-value
True Experiment under Non-Null Conditions 1000 Experiments under Null Conditions Distribution of Si is the NULL distribution p value = #(So > Si)/1000 No/few assumptions, but very expensive and time consuming vs Parametric: compute Null distribution from data; done.

17 Permutation Distribution of is the NULL distribution
True Experiment under Non-Null Conditions 1000 Permutations Distribution of is the NULL distribution p value = #(So > )/1000

18 Permutation: Shuffling and Sign Flipping
Can also do both Which method you use depends on the nature of the data One-sample group mean has to be done with sign flipping

19 Implementation usually done by changing X
= Swap Inputs 2 and 3 Swap Rows 2 and 3

20 Number of Permutations
N observations: N! possible shufflings 2N possible sign flips May be a huge number of possibilities N=100  shufflings Use a random subset of 1,000-10,000 Maybe smaller if using restricted exchanges

21 Why permutation works

22 Why Parametric Fails

23 Covariance Structure “Joint distribution”
Nobservations x Nobservations

24 Covariance Structure Nobservations x Nobservations
“iid” – independent, identically distributed

25 Reminder… Must have distribution of Si be that of the NULL.
True Experiment under Non-Null Conditions 1000 Permutations Must have distribution of Si be that of the NULL. When does that happen?

26

27 Limitations on permutation

28 What happens to Sn under permutation
Shuffle inputs 1 and 2 Flip sign of input 2. Diagonal not affected

29 “Exchangeability” – Joint distribution cannot change
Shuffle All subjects have same var Cov between subjects same Exchangeable Errors (EE) Compound Symmetry Sign Flip Subjects can have diff var Cov must = 0 Independent, Symmetric Errors (ISE) “Exchangeability” – Joint distribution cannot change

30 Exchangeability and Off-Diagonals
Caused by systematic variation in the residual, eg, if one input is above the mean, then another also tends to be above the mean In time series, correlation across time (temporal correlation) In cross-subject analysis, can be caused by continuous nuisance effects like age present under the NULL. Longitudinal Analysis, siblings, … Affects both shuffling (off-diagonals not equal) and sign flipping (off-diagonals not zero)

31 Restricted Exchangeability
Covariance matrix may have block-like structure Eg, repeated (two) measures Two subjects, A and B Matrix not exchangeable!

32 Exchangeability and Skew
Skew (3rd moment) indicates an asymmetric distribution Rayleigh, Poisson, Exponential, Gamma Eg, Income distribution Invalidates sign flipping, one sample group mean (see Eklund 2016) Statistical tests for skew Not a problem when computing a difference (eg, Group1 vs Group2)

33 Restricted Exchangeability
Covariance matrix may have block-like structure Eg, repeated (two) measures Two subjects, A and B Matrix not exchangeable! Block Exchange: Shuffle A and B blocks without changing order inside A or B Can become quite elaborate Reduces total number of possible permutations Winkler, et al, 2014 & 2015, Neuroimage.

34 Permutation with Nuisance Variables
Design matrix partitioning X – effects of interest Z – nuisance effects (eg, age) Many methods All approximate ter Braak – compute residuals from full model and permute residuals Freedman-Lane – recommended by Winkler 2014 Note: if age is a variable of interest, then you can just permute Winkler, et al, 2014, NI, Permutation Inference for the General Linear Model.

35 Computational Considerations
Need several thousand iterations People used to getting parametric results instantly May take a few hours May take a few days Parallelize (CPUs, GPUs) Early termination Tail approximations Winkler, et al, 2016, NI. Faster permutation in brain imaging

36 Parametric vs Permutation
Parametric Assumptions t-Test – Gaussian, independent, homoskedastic GRFT Cluster – Gaussian smoothness, smoothness level, cluster-forming threshold Parametric is Fast and Easy Permutation Makes many fewer assumptions Flexible – almost any quantity can be used, don’t need a closed-form solution of distribution Can be computationally intense Exchangeability is complicated May be a little less powerful than parametric when parametric assumptions are met

37 Summary Cluster Failure is driving interest in permutation
Shuffling and sign flipping (or both) Exchangeability - joint distribution cannot change Restricted exchange Nuisance variables Skew

38

Download ppt "Introduction to Permutation for Neuroimaging HST"

Similar presentations

SPM 2002 C1C2C3 X =  C1 C2 Xb L C1 L C2  C1 C2 Xb L C1  L C2 Y Xb e Space of X C1 C2 Xb Space X C1 C2 C1  C3 P C1C2  Xb Xb Space of X C1 C2 C1 

SPM 2002 C1C2C3 X =  C1 C2 Xb L C1 L C2  C1 C2 Xb L C1  L C2 Y Xb e Space of X C1 C2 Xb Space X C1 C2 C1  C3 P C1C2  Xb Xb Space of X C1 C2 C1 
Basics of fMRI Group Analysis Douglas N. Greve. 2 fMRI Analysis Overview Higher Level GLM First Level GLM Analysis First Level GLM Analysis Subject 3.

Basics of fMRI Group Analysis Douglas N. Greve. 2 fMRI Analysis Overview Higher Level GLM First Level GLM Analysis First Level GLM Analysis Subject 3.
Statistical Signal Processing for fMRI

Statistical Signal Processing for fMRI
Topological Inference Guillaume Flandin Wellcome Trust Centre for Neuroimaging University College London SPM Course London, May 2014 Many thanks to Justin.

Topological Inference Guillaume Flandin Wellcome Trust Centre for Neuroimaging University College London SPM Course London, May 2014 Many thanks to Justin.
Multiple testing Justin Chumbley Laboratory for Social and Neural Systems Research Institute for Empirical Research in Economics University of Zurich With.

Multiple testing Justin Chumbley Laboratory for Social and Neural Systems Research Institute for Empirical Research in Economics University of Zurich With.
07/01/15 MfD 2014 Xin You Tai & Misun Kim

07/01/15 MfD 2014 Xin You Tai & Misun Kim
Multiple testing Justin Chumbley Laboratory for Social and Neural Systems Research Institute for Empirical Research in Economics University of Zurich With.

Multiple testing Justin Chumbley Laboratory for Social and Neural Systems Research Institute for Empirical Research in Economics University of Zurich With.
Multiple comparison correction Methods & models for fMRI data analysis 18 March 2009 Klaas Enno Stephan Laboratory for Social and Neural Systems Research.

Multiple comparison correction Methods & models for fMRI data analysis 18 March 2009 Klaas Enno Stephan Laboratory for Social and Neural Systems Research.
Comparison of Parametric and Nonparametric Thresholding Methods for Small Group Analyses Thomas Nichols & Satoru Hayasaka Department of Biostatistics U.

Comparison of Parametric and Nonparametric Thresholding Methods for Small Group Analyses Thomas Nichols & Satoru Hayasaka Department of Biostatistics U.
Multiple comparison correction Methods & models for fMRI data analysis 29 October 2008 Klaas Enno Stephan Branco Weiss Laboratory (BWL) Institute for Empirical.

Multiple comparison correction Methods & models for fMRI data analysis 29 October 2008 Klaas Enno Stephan Branco Weiss Laboratory (BWL) Institute for Empirical.
1 Overview of Hierarchical Modeling Thomas Nichols, Ph.D. Assistant Professor Department of Biostatistics Mixed Effects.

1 Overview of Hierarchical Modeling Thomas Nichols, Ph.D. Assistant Professor Department of Biostatistics Mixed Effects.
Linear and generalised linear models Purpose of linear models Least-squares solution for linear models Analysis of diagnostics Exponential family and generalised.

Linear and generalised linear models Purpose of linear models Least-squares solution for linear models Analysis of diagnostics Exponential family and generalised.
General Linear Model & Classical Inference

General Linear Model & Classical Inference
General Linear Model & Classical Inference Guillaume Flandin Wellcome Trust Centre for Neuroimaging University College London SPM M/EEGCourse London, May.

General Linear Model & Classical Inference Guillaume Flandin Wellcome Trust Centre for Neuroimaging University College London SPM M/EEGCourse London, May.
2nd Level Analysis Jennifer Marchant & Tessa Dekker

2nd Level Analysis Jennifer Marchant & Tessa Dekker
Methods for Dummies Second level analysis

Methods for Dummies Second level analysis
Basics of fMRI Inference Douglas N. Greve. Overview Inference False Positives and False Negatives Problem of Multiple Comparisons Bonferroni Correction.

Basics of fMRI Inference Douglas N. Greve. Overview Inference False Positives and False Negatives Problem of Multiple Comparisons Bonferroni Correction.
Brain Mapping Unit The General Linear Model A Basic Introduction Roger Tait

Brain Mapping Unit The General Linear Model A Basic Introduction Roger Tait
7/16/2014Wednesday Yingying Wang

7/16/2014Wednesday Yingying Wang
SPM Course Zurich, February 2015 Group Analyses Guillaume Flandin Wellcome Trust Centre for Neuroimaging University College London With many thanks to.

SPM Course Zurich, February 2015 Group Analyses Guillaume Flandin Wellcome Trust Centre for Neuroimaging University College London With many thanks to.

Similar presentations

Ads by Google