1. Generative Models of M/EEG: Group inversion and MEG+EEG+fMRI multimodal integration Rik Henson (with much input from Karl Friston)

2. Overview 1.A Generative Model of M/EEG 2.Group inversion (optimising priors across subjects) 3.Multimodal integration: 3.1 Symmetric integration (fusion) of MEG + EEG 3.2 Asymmetric integration of MEG + fMRI 3.3 Full fusion of MEG/EEG + fMRI?

3. 1. A PEB Framework for MEG/EEG (Generative Model) Phillips et al (2005), Neuroimage Y = Data n sensors J = Sources p>>n sources L = Leadfieldsn sensors x p sources E = Error n sensors (Linear) Forward Model for MEG/EEG (for one timepoint): (Gaussian) Likelihood: C (e) = n x n Sensor (error) covariance Prior: C (j) = p x p Source (prior) covariance Posterior:

4. Specifying (co)variance components (priors/regularisation): 1. Sensor components, (error): C = Sensor/Source covariance Q = Covariance components λ = Hyper-parameters 2. Source components, (priors/regularisation): “IID” (white noise): # sensors Empty-room: # sensors “IID” (min norm): # sources Multiple Sparse Priors (MSP): # sources Friston et al (2008) Neuroimage 1. A PEB Framework for MEG/EEG (Generative Model)

5. Friston et al (2008) Neuroimage Fixed Variable Data

6. 1. A PEB Framework for MEG/EEG (Inversion) Friston et al (2002) Neuroimage 1. Obtain Restricted Maximum Likelihood (ReML) estimates of the hyperparameters (λ) by maximising the variational “free energy” (F): 2. Obtain Maximum A Posteriori (MAP) estimates of parameters (sources, J): cf. Tikhonov …and an estimate of their posterior covariance (inverse precision): 3. Maximal F approximates Bayesian (log) “model evidence” for a model, m: (relevant to MEG+EEG integration) (relevant to MEG+fMRI integration)

7. Summary: 1. A PEB Framework for MEG/EEG Automatically “regularises” in principled fashion… …allows for multiple constraints (priors)… …to the extent that multiple (100’s) of sparse priors possible… …(or multiple error components or multiple fMRI priors)… …furnishes estimates of source precisions and model evidence

8. 2. Group Inversion Specifying (co)variance components (priors/regularisation): 1. Sensor components, (error): C = Sensor/Source covariance Q = Covariance components λ = Hyper-parameters 2. Source components, (priors/regularisation): “IID” (white noise): # sensors Empty-room: # sensors “IID” (min norm): # sources Multiple Sparse Priors (MSP): # sources Friston et al (2008) Neuroimage

9. Specifying (co)variance components (priors/regularisation): 1. Sensor components, (error): C = Sensor/Source covariance Q = Covariance components λ = Hyper-parameters “IID” (white noise): # sensors Empty-room: # sensors 2. Optimise Multiple Sparse Priors by pooling across participants 2. Group Inversion Litvak & Friston (2008) Neuroimage # sources

10. 2. Group Inversion (single subject) (Generative Model) Litvak & Friston (2008) Neuroimage

11. 2. Group Inversion (multiple subjects) (Generative Model) Litvak & Friston (2008) Neuroimage

12. …projecting data and leadfields to a reference subject (0): Common source-level priors: Subject-specific sensor-level priors: 2. Group Inversion (Generative Model) Litvak & Friston (2008) Neuroimage

13. 2. Group Inversion (Generative Model) Litvak & Friston (2008) Neuroimage MMN MSP MSP (Group)

14. fMRIMEG? (future) Data: Causes (hidden): Generative (Forward) Models: Balloon Model Head Model ? EEG Head Model “Neural” Activity 3. Types of Multimodal Integration (inversion)

15. Asymmetric Integration fMRIMEG? (future) Data: Causes (hidden): Generative (Forward) Models: Balloon Model Head Model ? EEG Head Model “Neural” Activity Symmetric Integration (Fusion) 3. Types of Multimodal Integration Daunizeau et al (2007), Neuroimage

16. Specifying (co)variance components (priors/regularisation): 1. Sensor components, (error): C = Sensor/Source covariance Q = Covariance components λ = Hyper-parameters 2. Source components, (priors/regularisation): “IID” (white noise): # sensors Empty-room: # sensors “IID” (min norm): # sources Multiple Sparse Priors (MSP): # sources Friston et al (2008) Neuroimage 3.1 Fusion of MEG+EEG (Theory)

17. Specifying (co)variance components (priors/regularisation): 1. Sensor components, (error): C i (e) = Sensor error covariance for ith modality Q ij = jth component for ith modality λ ij = Hyper-parameters 2. Source components, (priors/regularisation): “IID” (min norm): # sources Multiple Sparse Priors (MSP): # sources # sensors E.g, white noise for 2 modalities: Henson et al (2009) Neuroimage 3.1 Fusion of MEG+EEG (Theory)

18. 3.1 Fusion of MEG+EEG (Generative Model) Henson et al (2009) Neuroimage

19. 3.1 Fusion of MEG+EEG (Generative Model) Henson et al (2009) Neuroimage

20. 3.1 Fusion of MEG+EEG (Theory) Henson et al (2009) Neuroimage Stack data and leadfields for d modalities: Where data / leadfields scaled to have same average / predicted variance: m i = Number of spatial modes (e.g, channels) (note: common sources and source priors, but separate error components)

21. ERs from 12 subjects for 3 simultaneously-acquired Neuromag sensor-types: RMS fT/m VV Faces Scrambled fT 3.1 Fusion of MEG+EEG (Application) Magnetometers (MEG, 102) (Planar) Gradiometers (MEG, 204) Electrodes (EEG, 70) Henson et al (2009) Neuroimage 150-190ms Faces - Scrambled ms

22. MEG mags MEG grads EEG FUSED +31 -51 -15 +19 -48 -6 +43 -67 -11 +44 -64 -4 3.1 Fusion of MEG+EEG Henson et al (2009) Neuroimage IID noise for each modality; common MSP for sources (fixed number of spatial+temporal modes) Scrambled 150-190msFaces – Scrambled, Faces

23. 3.1 Fusion of MEG+EEG (Conclusions) Henson et al (2009) Neuroimage Fusing magnetometers, gradiometers and EEG increased the conditional precision of the source estimates relative to inverting any one modality alone (when equating number of spatial+temporal modes) The maximal sources recovered from fusion were a plausible combination of the ventral temporal sources recovered by MEG and the lateral temporal sources recovered by EEG (Simulations show the relative scaling of mags and grads agrees with empty-room data)

24. 3.2 Integration of M/EEG+fMRI Specifying (co)variance components (priors/regularisation): 1. Sensor components, (error): C = Sensor/Source covariance Q = Covariance components λ = Hyper-parameters 2. Source components, (priors/regularisation): “IID” (white noise): # sensors Empty-room: # sensors “IID” (min norm): # sources Multiple Sparse Priors (MSP): # sources Friston et al (2008) Neuroimage

25. Henson et al (in press) Human Brain Mapping Specifying (co)variance components (priors/regularisation): 1. Sensor components, (error): C = Sensor/Source covariance Q = Covariance components λ = Hyper-parameters “IID” (white noise): # sensors Empty-room: # sensors “IID” (min norm): # sources fMRI Priors: # sources 2. Each suprathreshold fMRI cluster becomes a separate prior 3.2 Integration of M/EEG+fMRI

26. 3.2 Integration of M/EEG+fMRI (Generative Model)

28. Henson et al (in press) Human Brain Mapping T1-weighted MRI Anatomical data {T,F,Z}-SPM Gray matter segmentation Cortical surface extraction 3D geodesic Voronoï diagram Functional data … 1. Thresholding and connected component labelling … 2. Projection onto the cortical surface using the Voronoï diagram … 3. Prior covariance components 3.2 Integration of M/EEG+fMRI (Priors)

29. 3.2 Integration of M/EEG+fMRI (Application) Henson et al (in press) Human Brain Mapping Prior 4.Prior 5. SPM{F} for faces versus scrambled faces, 15 voxels, p<.05 FWE 5 clusters from SPM of fMRI data from separate group of (18) subjects in MNI space 12 3 45

30. 3.2 Fusion of MEG+fMRI (Application) Henson et al (in press) Human Brain Mapping Prior 4.Prior 5. (binarised, variance priors) Magnetometers (MEG) * * * * None Global Local (Valid) Local (Invalid) Valid+Invalid Electrodes (EEG) Negative Free Energy (a.u.) (model evidence) * * * * * * * Gradiometers (MEG)

31. 3.2 Fusion of MEG+fMRI (Application) Henson et al (in press) Human Brain Mapping Prior 4.Prior 5. (binarised, variance priors) Magnetometers (MEG) * * * * Gradiometers (MEG) None Global Local (Valid) Local (Invalid) Valid+Invalid Electrodes (EEG) Negative Free Energy (a.u.) (model evidence) * * * * * * *

32. 3.2 Fusion of MEG+fMRI (Application) Henson et al (in press) Human Brain Mapping Prior 4.Prior 5. (binarised, variance priors) Magnetometers (MEG) * * * * Gradiometers (MEG) None Global Local (Valid) Local (Invalid) Valid+Invalid Electrodes (EEG) Negative Free Energy (a.u.) (model evidence) * * * * * * *

33. 3.2 Fusion of MEG+fMRI (Application) Henson et al (in press) Human Brain Mapping Prior 4.Prior 5. (binarised, variance priors) Magnetometers (MEG) * * * * Gradiometers (MEG) None Global Local (Valid) Local (Invalid) Valid+Invalid Electrodes (EEG) Negative Free Energy (a.u.) (model evidence) * * * * * * *

34. 3.2 Fusion of MEG+fMRI (Application) Henson et al (in press) Human Brain Mapping Prior 4.Prior 5. (binarised, variance priors) Magnetometers (MEG) * * * * Gradiometers (MEG) None Global Local (Valid) Local (Invalid) Valid+Invalid Electrodes (EEG) Negative Free Energy (a.u.) (model evidence) * * * * * * *

35. 3.2 Fusion of MEG+fMRI (Application) Henson et al (in press) Human Brain Mapping None Global Local (Valid) Local (Invalid) Magnetometers (MEG) Gradiometers (MEG) Electrodes (EEG) IID sources and IID noise (L2 MNM)

36. 3.2 Fusion of MEG+fMRI (Application) Henson et al (in press) Human Brain Mapping None Global Local (Valid) Local (Invalid) Magnetometers (MEG) Gradiometers (MEG) Electrodes (EEG) IID sources and IID noise (L2 MNM)

37. 3.2 Fusion of MEG+fMRI (Application) Henson et al (in press) Human Brain Mapping fMRI priors counteract superficial bias of L2-norm None Global Local (Valid) Local (Invalid) Magnetometers (MEG) Gradiometers (MEG) Electrodes (EEG) IID sources and IID noise (L2 MNM)

38. 3.2 Fusion of MEG+fMRI (Application) Henson et al (in press) Human Brain Mapping fMRI priors counteract superficial bias of L2-norm None Global Local (Valid) Local (Invalid) Magnetometers (MEG) Gradiometers (MEG) Electrodes (EEG) IID sources and IID noise (L2 MNM)

39. 3.2 Fusion of MEG+fMRI (Application) Henson et al (in press) Human Brain Mapping Prior 4.Prior 5. NB: Priors affect variance, not precise timecourse… R L Gradiometers (MEG) (5 Local Valid Priors) Differential Response (Faces vs Scrambled) Differential Response (Faces vs Scrambled) Right Posterior Fusiform (rPF) Right Medial Fusiform (rMF) Right Lateral Fusiform (rLF) Left occipital pole (lOP) -27 -93 0 +26 -76 -11 +41 -43 -24 +32 -45 -12 -43 -47 -21 Left Lateral Fusiform (lLF) Differential Response (Faces vs Scrambled)

40. Adding a single, global fMRI prior increases model evidence Adding multiple valid priors increases model evidence further Helpful if some fMRI regions produce no MEG/EEG signal (or arise from neural activity at different times) Adding invalid priors rarely increases model evidence, particularly in conjunction with valid priors Can counteract superficial bias of, e.g, minimum-norm Affects variance but not not precise timecourse (Adding fMRI priors to MSP has less effect) 3.2 Fusion of MEG+fMRI (Conclusions) Henson et al (in press) Human Brain Mapping

41. 3.3 Fusion of fMRI and MEG/EEG? fMRIMEG? (future) Data: Causes (hidden): Balloon Model Head Model ? EEG Head Model “Neural” Activity Fusion of fMRI + MEG/EEG? Henson (2010) Biomag

42. 3.3 Fusion of fMRI and MEG/EEG? Henson (2010) Biomag

43. time (t)? space (s) 3.3 Fusion of fMRI and MEG/EEG? Henson (2010) Biomag

44. Overall Conclusions 1.The PEB (in SPM8) framework is advantageous 2.Group optimisation of MSPs can be advantageous 3.Full fusion of MEG and EEG is advantageous 4.Using fMRI as (spatial) priors on MEG is advantageous 5.Unclear that fusion of fMRI and M/EEG is advantageous

45. The End

46. 3. Fusion of MEG+EEG Henson et al (2009) Neuroimage

47. Participant Grads Mags log(λ x 10 6 ) Participant EEG Grads Mags log(λ x 10 6 ) 3. Fusion of MEG+EEG Hyperparameters Henson et al (2009) Neuroimage

48. 4. Fusion of MEG+fMRI Henson et al (in press) Human Brain Mapping Prior 4.Prior 5. fMRI hyperparameters ln(λ)+32 Participant Magnetometers (MEG)Gradiometers (MEG)Electrodes (EEG) Local Valid Local Invalid