1 Data Analysis for fMRI Computational Analyses of Brain Imaging CALD and Psychology Tom M. Mitchell and Marcel Just January 15, 2003

2 Ten Minutes of Activity for One Voxel Indicates experimental condition

3

4 …

5 fMRI Data Visualization [from W. Schneider] Slice View Time Series 3 D view Rendered View Inflated View

6 Many Types of Analysis Transformation from fourier space into spatial images, adjusting for head motion, noise, drift,... (FIASCO, SVM) Warping individual brains to canonical structure (Talairach, AIR, SPM) Identifying voxels activated during task (t-test, F-test,…) Finding temporally correlated voxels (clustering) Factoring signal into few components (PCA, ICA) Modeling temporal evolution of activity (diffeqs, HMMs) Learning classifiers to detect cognitive states (Bayes, SVM) Modeling higher cognitive processes (4CAPS, ACT-R) Combining fMRI with ERP, behavioral data, …

7 Identifying Voxels Activated During Task For each voxel, v i, calculate t statistic comparing activity of v i during task versus rest condition. Retain voxels with t-statistic above some threshold

8 Mental Rotation of Imagined Objects Clock rotation Shephard-Metz rotation both [Just, et al., 2001]

9 “Men listen with only one side of their brains, while women use both” (IU School of Medicine Department of Radiology)  Men listening  Women listening Study of Men and Women Listening

10 Identifying Voxels with Similar Time Courses (functional connectivity)

11 The activation in two cortical areas (parietal/dorsal and inferior temporal/ventral) becomes more synchronized as the object recognition task becomes more difficult. Easier Harder Increase in functional connectivity between parietal and inferior temporal areas with workload (from Diwadkar, Carpenter, & Just, 2001) Figure 7

12 Factoring fMRI Signals into Fewer Components PCA, ICA, SVD, Hidden Units

13 Independent Component Analysis of fMRI time-series ICA discovers statistically independent components that combine to form the observed fMRI signal ICA is a data-driven approach, complementary to hypothesis-driven methods (e.g. GLM) for analyzing fMRI data Finds reduced dimensionality descriptions of poorly understood, high dimensional spaces Requires no a-priori knowledge about hemodynamics, noise models, time-courses of subject stimuli,…

14 (McKeown et al., 1998) Independent Component Analysis of fMRI time-series: data-model

15 fMRI time-series ICA algorithm IC #1 IC #2 IC #T Independent Component Analysis of fMRI Time-series [from W. Schneider]

16 ICA Solution IC1 IC2 Independent Component Analysis of fMRI time-series GLM Solution Images Elia Formisano & Rainer Goebel 2001

17 Advantages of ICA Interpretation of non-explicit condition manipulation –Not just AB type designs –Applications driving, reading, problem solving Identify dimensions of poorly understood spaces –Reduce high dimension data to few components –Applications: structure of semantic memory, processes underlying visual scene analysis in visual cortex

18 Learning Classifiers to Decode Cognitive States from fMRI Bayes classifiers, SVM’s, kNN, …

19 Study 1: Word Categories Family members Occupations Tools Kitchen items Dwellings Building parts 4 legged animals Fish Trees Flowers Fruits Vegetables [Francisco Pereira et al.]

20 Training Classifier for Word Categories Learn fMRI(t)  word-category(t) –fMRI(t) = 8470 to 11,136 voxels, depending on subject Feature selection: Select n voxels –Best single-voxel classifiers –Strongest contrast between fixation and some word category –Strongest contrast, spread equally over ROI’s –Randomly Training method: –train ten single-subect classifiers –Gaussian Naïve Bayes  P(fMRI(t) | word-category)

21 Results Classifier outputs ranked list of classes Evaluate by the fraction of classes ranked ahead of true class 0=perfect, 0.5=random, 1.0 unbelievably poor

22 Impact of Feature Selection

23

24 Summary Able to classify instantaneous cognitive state –in contrast to describing average activity over time Significance –Virtual sensors for mental states –Step toward modeling sequential cognitive processes? –Potential clinical applications: diagnosis = classification

25 Modeling temporal evolution of activity HMMs, Diffeqs, …

26 Challenge: learn process model -- HMM’s? a=6,… 3x+a=2 recall correctrecall erroranswer transform correct transform error read problem time  start …

27 V1 V4 BA37 STG BA39 Perturbing inputs Stimuli-bound u 1 (t) {e.g. visual words} DCM [Friston 2002] Aim: Functional integration and the modulation of specific pathways yyyyy Contextual inputs Stimulus-free - u 2 (t) {e.g. cognitive set/time}

28 yyy Hemodynamic model The DCM and its bilinear approximation [Friston 02] Input u(t) activity x 1 (t) activity x 3 (t) activity x 2 (t) The bilinear model neuronal changes intrinsic connectivity induced response induced connectivity

29 Constraints on Connections Hemodynamic parameters Applications Simulations Plasticity in single word processing Attentional modulation of coupling Models of Hemodynamics in a single region Neuronal interactions Overview [Friston, 2002] Bayesian estimation

30 Cognitive Models Grounded in fMRI Data

31 4CAPS Model of Language Processing [Just, et al., 2002]

32 The player was followed by the parent. [Just, et al., 2002]

33 4 4CAPS Prediction offMRIActivity Figure 10 Model CU transform CU in 4CAPS comprehension model components fMRI data Model prediction

34 [Anderson, Qin, & Sohn, 2002]

35 Cognitive model: Observed image sequence: See word Recognize word Understand statement Answer question Hypothesized intermediate states, representations, processes: time  Understand question What We’d Like

36 Machine Learning Problems Learn f: image(t)  cognitiveState(t) Discover useful intermediate abstractions Learn process models