1. SCANLab http://www.columbia.edu/cu/psychology/tor/ Meta-analysis of neuroimaging data What, Why, and How Tor D. Wager Columbia University

2. SCANLab http://www.columbia.edu/cu/psychology/tor/ Uses of meta-analysis in neuroimaging Meta-analysis is an essential tool for summarizing the vast and growing neuroimaging literature Wager, Lindquist, & Hernandez, in press

3. SCANLab http://www.columbia.edu/cu/psychology/tor/ Uses of meta-analysis in neuroimaging Wager, Lindquist, & Kapan, 2007 Assess consistency of activation across laboratories and task variants Compare across many types of tasks and evaluate the specificity of activated regions for particular psychological conditions Identify and define boundaries of functional regions Co-activation: Develop models of functional systems and pathways

4. SCANLab http://www.columbia.edu/cu/psychology/tor/ Functional networks in meta-analysis Use regions or distributed networks in a priori tests in future studies

5. SCANLab http://www.columbia.edu/cu/psychology/tor/ Meta-analyses of cognitive control

6. SCANLab http://www.columbia.edu/cu/psychology/tor/ Meta-analyses of emotion & motivation

7. SCANLab http://www.columbia.edu/cu/psychology/tor/ Meta-analyses of disorders

8. SCANLab http://www.columbia.edu/cu/psychology/tor/ Meta-analyses of language

9. SCANLab http://www.columbia.edu/cu/psychology/tor/ Meta-analyses of other stuff

10. SCANLab http://www.columbia.edu/cu/psychology/tor/ Using meta-analysis to evaluate consistency: Why?

11. SCANLab http://www.columbia.edu/cu/psychology/tor/ Locating emotion-responsive regions 164 PET/fMRI studies, 437 activation maps, 2478 coordinates

12. SCANLab http://www.columbia.edu/cu/psychology/tor/ Why identify consistent areas? Making statistic maps in neuroimaging studies involves many tests (~100,000 per brain map) Many studies use uncorrected or improperly corrected p-values Long-term Memory P-value thresholds used Corr. # of Maps Uncorrected How many false positives? A rough estimate: 663 peaks, 17% of reported activations Wager, Lindquist, & Kaplan, 2007

13. SCANLab http://www.columbia.edu/cu/psychology/tor/ Consistency Reported peaks 163 studies Consistently Activated regions Emotion: 163 studies

14. SCANLab http://www.columbia.edu/cu/psychology/tor/ mTC pOFC vmPFC BF aINS sTC latOFC lFG TC pgACC rdACC dmPFC PCC OCC sgACC vmPFC CM, MD Deep nuclei Gyrus rectus Central sulcus dmPFC Pre SMA Fig 4: MKDA Results Ventral surface Lateral surface (R)Medial surface (L) Kober et al., in press, NI

15. SCANLab http://www.columbia.edu/cu/psychology/tor/ Using meta-analysis to evaluate specificity: Why?

16. SCANLab http://www.columbia.edu/cu/psychology/tor/ Disgust responses: Specificity in insula? Insula

17. SCANLab http://www.columbia.edu/cu/psychology/tor/ Disgust responses: Specificity in insula? Feldman-Barrett & Wager, 2005; Phan, Wager, Taylor, & Liberzon, 2002; Phan, Wager, Liberzon & Taylor, 2004 Search Area: Insula

18. SCANLab http://www.columbia.edu/cu/psychology/tor/ Meta-analysis plays a unique role in answering… Is it reliable? –Would each activated region replicate in future studies? –Would activation be insensitive to minor variations in task design? Is it task-specific? –Predictive of a particular psychological state or task type? –Diagnostic value? The Neural Correlates of Task X

19. SCANLab http://www.columbia.edu/cu/psychology/tor/ Using meta-analysis to evaluate consistency: How?

20. SCANLab http://www.columbia.edu/cu/psychology/tor/ Monte Carlo: Expected maximum proportion Under the null hypothesis Apply threshold Weighted average E Damasio, 2000Liberzon, 2000 Wicker, 2003 Peak coordinate locations (437 maps) … Kernel convolution Comparison indicator maps … Proportion of activated Comparisons map (from 437 comparisons) Significant regions Meta-analysis: Multilevel kernel density estimate (MKDE) Wager, Lindquist, & Kaplan, 2007; Etkin & Wager, in press Permute blobswithin studymaps

21. SCANLab http://www.columbia.edu/cu/psychology/tor/ MKDA: Key points Statistic reflects consistency across studies. Study comparison map is treated as a random effect. Peaks from one study cannot dominate. Studies are weighted by quality (see additional info on handouts for rationale) Spatial covariance is preserved in Monte Carlo. Less sensitive to arbitrary standards for how many peaks to report.

22. SCANLab http://www.columbia.edu/cu/psychology/tor/ Whether and how to weight studies/peaks MKDA analysis weights by sqrt(sample size) and study quality (including fixed/random effects) Fixed effects Random effects Activation indicator (1 or 0) for map c Study quality weight Sample size for map c Weighted proportion of activating studies Weighted average

23. SCANLab http://www.columbia.edu/cu/psychology/tor/ Monte Carlo Simulation Simulation vs. theory (e.g. Poisson process) Simulation allows: –Non-stationary spatial distribution of peaks (clumps) under null hypothesis; randomize blob locations –Family-wise error rate control with irregular (brain-shaped) search volume –Cluster size inference, given primary threshold Monte Carlo: E(max(P|H 0 ))

24. SCANLab http://www.columbia.edu/cu/psychology/tor/ Compare with Activation Likelihood Estimate (ALE), Kernel Density Analysis (KDA) Peak coordinates Combined across studies Kernel convolution Density kernel ALE kernel OR Peak density or ALE map Apply significance threshold Significant results Density kernel: Chein, 1998; Phan et al., 2002; Wager et al., 2003, 2004, 2007, in press Gaussian density kernel + ALE: Turkeltaub et al., 2002; Laird et al., 2005; others Ignores the fact that some studies report more peaks than others!

25. SCANLab http://www.columbia.edu/cu/psychology/tor/ Comparison with other methods Statistic reflects consistency across studies. Study comparison map is treated as a random effect. Peaks from one study cannot dominate. Studies are weighted by quality Spatial covariance is preserved in Monte Carlo. Less sensitive to arbitrary standards for how many peaks to report. Peaks are lumped together, study is fixed effect. Peaks from one study can dominate, studies that report more peaks dominate. No weighting, or z-score weighting (problematic) Spatial covariance is not preserved in Monte Carlo. Effects of reporting standards large. MKDA KDA/ALE See handouts for more comparison points

26. SCANLab http://www.columbia.edu/cu/psychology/tor/ ALE approach Treats points as if they were Gaussian probability distributions. Summarize the union of probabilities at each voxel: probability of any peak “truly” lying in that voxel is the probability that peak X i lies in a given voxel The bar indicates the complement operator Null hypothesis:No peaks lie in voxel Alt hypothesis:At least one peak lies in voxel

27. SCANLab http://www.columbia.edu/cu/psychology/tor/ ALE meta-analysis Analyst chooses smoothing kernel ALE analysis with zero smoothing: –Every voxel reported in any study is significant in the meta-analysis Test case: 3-peak meta analysis, one peak activates in voxel: ALE statistic: Highest possible value! In practice: 10 – 15 mm FWHM kernel

28. SCANLab http://www.columbia.edu/cu/psychology/tor/ Comparison across methods: Inference PropertyKDAALEMultilevel KDA KernelSphericalGaussianSpherical Interpretation of statistic Num nearby peaksProb. that at least one peak nearby Num. study maps activating nearby Null hypothesisPeaks are not spatially consistent No peaks truly activate Study maps are not spatially consistent Interpretation of significant result More peaks lie near voxel than expected by chance One or more peaks lies at this voxel A higher proportion of studies activate near voxel than expected by chance Assumptions1. Study is fixed effect (homogenous sample of studies) 2. Peaks are spatially independent under the null hypothesis 1. Study is fixed effect (homogenous sample of studies) 2. Peaks are spatially independent under the null hypothesis Activation ‘blobs’ are spatially independent under the null hypothesis Generalize toNew peaks from same studies New study maps

29. SCANLab http://www.columbia.edu/cu/psychology/tor/ Comparison: Correction and Weighting PropertyKDAALEMultilevel KDA Multiple comparisons FWERFDRFWER (recommended) or FDR WeightingNone, or weight peaks by z-score NoneWeight studies by sample size, fixed/random effects, quality

30. SCANLab http://www.columbia.edu/cu/psychology/tor/ Density analysis: Summary Working memory Executive WM Long-term memory InhibitionTask switching Memory Response selection Wager et al., 2004; Nee, Wager, & Jonides, 2007; Wager et al., in press; Van Snellenberg & Wager, in press

31. SCANLab http://www.columbia.edu/cu/psychology/tor/ Using meta-analysis to evaluate specificity: How?

32. SCANLab http://www.columbia.edu/cu/psychology/tor/ Specificity Task-related differences in relative activation frequency across the brain: –MKDA difference maps (e.g., Wager et al., 2008) Task-related differences in absolute activation frequency –Nonparametric chi-square maps (Wager, Lindquist, & Kaplan, 2007) Classifier systems to predict task type from distributed patterns of peaks (e.g., Gilbert)

33. SCANLab http://www.columbia.edu/cu/psychology/tor/ MKDA Difference maps: Emotion example Approach: –Calculate density maps for two conditions, subtract to get difference maps –Monte Carlo: Randomize blob locations within each study, re- calculate density difference maps and save max –Repeat for many (e.g., 10,000) iterations to get max distribution –Threshold based on Monte Carlo simulation ExperiencedPerceived

34. SCANLab http://www.columbia.edu/cu/psychology/tor/ Emotion example: Selective regions AmyTPOFC Amy aIns Experience > Perception aIns OFC vaIns dmPFC Hy vaIns TP PAG Midb mOFC TP OFC Midb TP Hy Perception > Experience pgACC Amy CB IFG Amy CB D Wager et al., in press, Handbook of Emotion

35. SCANLab http://www.columbia.edu/cu/psychology/tor/ Task-brain activity associations in meta-analysis Study contrast map Region/V oxel 1 Task condition Study 11Disgust Study 20Fear Study 31Disgust Study 41Happiness Study 50Anger ……… Study N0Sadness Measures of association: Chi-square But requires high expected counts (> 5) in each cell. Not appropriate for map-wise testing over many voxels Fisher’s exact test (2 categories only) Multinomial exact test Computationally impractical! Nonparametric chi-square Approximation to exact test OK for low expected counts

36. SCANLab http://www.columbia.edu/cu/psychology/tor/ Nonparametric chi-square: Details Study contrast map Region/V oxel 1 Task condition Study 11Disgust Study 20Fear Study 31Disgust Study 41Happiness Study 50Anger ……… Study N0Sadness Idea of exact test: Conditionalize on marginal counts for activation and task conditions. Null hypothesis: no systematic association between activation and task P-value is proportion of null- hypothesis possible arrangements that can produce distribution across task conditions as large as observed or larger.

37. SCANLab http://www.columbia.edu/cu/psychology/tor/ Nonparametric chi-square: Details Study contrast map Region/V oxel 1 Task condition Study 1 0 Disgust Study 2 1 Fear Study 3 0 Disgust Study 4 1 Happiness Study 5 0 Anger … … … Study N 1 Sadness Permutation test: Permute activation indicator vector, creating null-hypothesis data (no systematic association) Marginal counts are preserved. Test 5,000 or more samples and calculate P-value based on observed null-hypothesis distribution

38. SCANLab http://www.columbia.edu/cu/psychology/tor/ Density difference vs. Chi-square Relative vs. absolute differences Voxels (one-dimensional brain) Experience Perception Chi-square Density

39. SCANLab http://www.columbia.edu/cu/psychology/tor/ Can we predict the emotion from the pattern of brain activity? Approach: predict studies based on their pattern of reported peaks (e.g., Gilbert, 2006) Use naïve Bayesian classifier (see work by Laconte; Tong; Norman; Haxby). Cross-validate: predict emotion type for new studies that are not part of training set. ExperiencedPerceived

40. SCANLab http://www.columbia.edu/cu/psychology/tor/ Classifying experienced emotion vs. perceived emotion: 80% accurate Experience Perception PAG vs. Ant. thalamus Deep cerebellar nuc. vs. Lat. cerebellum DMPFC vs. Pre-SMA EXP vs. PER DMPFC EXP PAG Deep cerebellar nuc.

41. SCANLab http://www.columbia.edu/cu/psychology/tor/ Outline: Why and How… Consistency: Replicability across studies –Consistency in single-region results: MKDA –Consistency in functional networks: MKDA + Co-activation Specificity and “reverse inference” –Brain-activity – psychological category mappings for individual brain regions: MKDA difference maps; Nonparametric Chi-square –Brain-activity – psychological category mappings for distributed networks Applying classifier systems to meta-analytic data

42. SCANLab http://www.columbia.edu/cu/psychology/tor/ Extending meta-analysis to connectivity Study contrast map Region/V oxel 1 Region/V oxel 2 Study 110 Study 200 Study 311 Study 411 Study 500 ……… Study N01 Co-activation: If a study (contrast map) activates within k mm of voxel 1, is it more likely to also activate within k mm of voxel 2? Measures of association: Kendall’s Tau-b Fisher’s exact test Nonparametric chi-square Others… N = 45Region 1 No Region 1 Yes Region 2 Yes 623 Region 2 No 124

43. SCANLab http://www.columbia.edu/cu/psychology/tor/ Kendall’s Tau: Details Ordinal “nonparametric” association between two variables, x and y Uses ranks; no assumption of linearity or normal distribution (Kendall, 1938, Biometrika) Values between [-1 to 1], like Pearson’s correlation Tau is proportion of concordant pairs of observations sign(x diff. between pairs)= sign(y diff. between pairs) Tau = (# concordant pairs - # discordant pairs) / total # pairs

44. SCANLab http://www.columbia.edu/cu/psychology/tor/ Meta-analysis functional networks: Examples Emotion: Kober et al. (in press), 437 maps

45. SCANLab http://www.columbia.edu/cu/psychology/tor/ Acknowledgements Ed Smith Funding agencies: National Science Foundation National Institute of Mental Health Martin Lindquist Derek Nee John Jonides Ed Smith Tom Nichols Lisa Feldman Barrett Hedy Kober Lauren Kaplan Jason Buhle Jared Van Snellenberg Luan Phan Steve Taylor Israel Liberzon Meta-analysis of emotion Statistics Meta-analysis of cognitive function Students

46. SCANLab http://www.columbia.edu/cu/psychology/tor/ Weighting

47. SCANLab http://www.columbia.edu/cu/psychology/tor/ Whether and how to weight studies/peaks Studies (and peaks) differ in sample size, methodology, analysis type, smoothness, etc. Advantageous to give more weight to more reliable studies/peaks Z-score weighting –Advantages: Weights nominally more reliable peaks more heavily –Disadvantages: Small studies can produce variable results. Reporting bias: High z-score peaks are high partially due to error; “capitalizing on chance” Must convert to common Z-score metric across different analysis types in different studies

48. SCANLab http://www.columbia.edu/cu/psychology/tor/ Whether and how to weight studies/peaks Alternative: Sample-size weighting –Advantages: Weights studies by the quality of information their peaks are likely to reflect Avoids overweighting peaks reported due to “capitalizing on chance” –Disadvantages: Ignores relative reliability of various peaks within studies

49. SCANLab http://www.columbia.edu/cu/psychology/tor/ MKDA vs. KDA vs. ALE: Comparison chart

50. SCANLab http://www.columbia.edu/cu/psychology/tor/ More details on reverse inference

51. SCANLab http://www.columbia.edu/cu/psychology/tor/ Is brain activity diagnostic of a particular psychological state? ‘Forward’ and ‘reverse’ inference are not the same! Reverse inference requires comparing across many psychological states! Pleasure? Punishing wrongdoers Brain activity Given a psychological state We observe brain activity P(Brain | Psy) Forward inference Can we infer psychological pleasure? P(Psy | Brain) Given brain activity Reverse inference

52. SCANLab http://www.columbia.edu/cu/psychology/tor/ The predictive value problem: Worked example For a brain region to be used as a marker of pleasure –The brain region must respond consistently to pleasure – The brain region must respond specifically to pleasure (not activated by other things) Ventral caudatePleasure P(Brain|Pleasure) =.9 Forward inference; Sensitivity Non-pleasure P(Brain|no pleasure) =.4 1-Specificity P(pleasure) =.1 Prior Caculate reverse inference: P(Pleasure|Brain) =.2

53. SCANLab http://www.columbia.edu/cu/psychology/tor/ More details on connectivity

54. SCANLab http://www.columbia.edu/cu/psychology/tor/ More details on MKDA difference maps and nonparametric chi-square maps