1. 1 Preliminary Experiments: Learning Virtual Sensors Machine learning approach: train classifiers –fMRI(t, t+  )  CognitiveState Fixed set of possible states Trained per subject, per experiment Time interval specified

2. 2 Approach Learn fMRI(t,…,t+k)  CognitiveState Classifiers: –Gaussian Naïve Bayes, SVM, kNN Feature selection/abstraction –Select subset of voxels (by signal, by anatomy) –Select subinterval of time –Average activities over space, time –Normalize voxel activities –…–…

3. 3 Study 1: Word Categories Family members Occupations Tools Kitchen items Dwellings Building parts 4 legged animals Fish Trees Flowers Fruits Vegetables [Francisco Pereira]

4. 4 Word Categories Study Ten neurologically normal subjects Stimulus: –12 blocks of words: Category name (2 sec) Word (400 msec), Blank screen (1200 msec); answer … –Subject answers whether each word in category –32 words per block, nearly all in category –Category blocks interspersed with 5 fixation blocks

5. 5 Training Classifier for Word Categories Learn fMRI(t)  word-category(t) –fMRI(t) = 8470 to 11,136 voxels, depending on subject Feature selection: Select n voxels –Best single-voxel classifiers –Strongest contrast between fixation and some word category –Strongest contrast, spread equally over ROI’s –Randomly Training method: –train ten single-subect classifiers –Gaussian Naïve Bayes  P(fMRI(t) | word-category)

6. 6 Learned Bayes Models - Means P(BrainActivity | WordCategory = People)

7. 7 Learned Bayes Models - Means P(BrainActivity | WordClass) Animal words People words Accuracy: 85%

8. 8 Results Classifier outputs ranked list of classes Evaluate by the fraction of classes ranked ahead of true class –0=perfect, 0.5=random, 1.0 unbelievably poor Try abstracting 12 categories to 6 categories e.g., combine “Family Members” with “Occupations”

9. 9 Impact of Feature Selection

10. 10 X 1 =S 1 +N 1 X 2 =S 2 +N 2 N0N0 Goal: learn f: X ! Y or P(Y|X) Given: 1.Training examples where X i = S i + N i, signal S i ~ P(S|Y=i), noise N i ~ P noise 2.Observed noise with zero signal N 0 ~ P noise “Zero Signal” learning setting. (fixation) Class 1 observations Class 2 observations

11. 11

12. 12 [Haxby et al., 2001]

13. 13 Study 1: Summary Able to classify single fMRI image by word category block Feature selection important Is classifier learning word category or something else related to time? –Accurate across ten subjects –Relevant voxels in similar locations across subjs –Locations compatible with earlier studies –New experimental data will answer definitively

14. 14 Study 2: Pictures and Sentences Trial: read sentence, view picture, answer whether sentence describes picture Picture presented first in half of trials, sentence first in other half Image every 500 msec 12 normal subjects Three possible objects: star, dollar, plus Collected by Just et al. [Xuerui Wang and Stefan Niculescu]

15. 15 It is true that the star is above the plus?

16. 16

17. 17 + --- *

18. 18

19. 19 Is Subject Viewing Picture or Sentence? Learn fMRI(t, …, t+15)  {Picture, Sentence} –40 training trials (40 pictures and 40 sentences) –7 ROIs Training methods: –K Nearest Neighbor –Support Vector Machine –Naïve Bayes

20. 20 Is Subject Viewing Picture or Sentence? Support Vector Machine worked better on avg. Results (leave one out) on picture-then- sentence, sentence-then-picture data –Random guess = 50% accuracy –SVM using pair of time slices at 5.0,5.5 sec after stimulus: 91% accuracy

21. 21 Accuracy for Single-Subject Classifiers

22. 22 Can We Train Subject-Indep Classifiers?

23. 23 Training Cross-Subject Classifiers Approach: define supervoxels based on anatomically defined regions of interest –Abstract to seven brain region supervoxels Train on n-1 subjects, test on nth

24. 24 Accuracy for Cross-Subject GNB Classifier

25. 25 Accuracy for Cross-Subject and Cross-Context Classifier

26. 26 Possible ANN to discover intermediate data abstraction across multiple subjects. Each bank of inputs corresponds to voxel inputs for a particular subject. The trained hidden layer will provide a low-dimensional data abstraction shared by all subjects. We propose to develop new algorithms to train such networks to discover multi-subject classifiers. Subject 1 Subject 2 Subject 3 Learned cross-subject representation Output classification