3rd EEGLAB Workshop Singapore Mining Event-Related Brain Dynamics Scott Makeig Swartz Center for Computational Neuroscience, Institute for Neural Computation, UCSD La Jolla CA Scott Makeig Event-Related Brain Dynamics I

EEGLAB An open-source EEG/MEG signal processing environment for Matlab Software for applying most of the techniques covered in this talk are freely available in the EEGLAB open source toolbox for Matlab. EEGLAB consists of several hundred Matlab functions, integrated under a graphic user interface by Arnaud Delorme (pictured). A growing list of plug-in capabilities have been contributed or published by other groups, making EEGLAB an attractive open source environment for exploratory signal processing of EEG, MEG, or other electrophysiological data. http://sccn.ucsd.edu/eeglab Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I EEGLAB Workshop 06 USA Netherlands Singapore Malaysia Taiwan Japan Australia South Korea United Arab Emirates Germany Italy England Israel Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Who Am I? Cortical macrodynamics Limitations of response averaging A richer model Independent component analysis Time/frequency analysis Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I I gaped … Who am I? I tossed … I held … I jumped ... I ducked I swerved … I reached … I threw …. I ran … I shot … Our most personal conception of our self is via experience, recognition and memory of cognitive events either involving purposeful motor actions … I pointed … I smiled … Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I I realized that … ? It struck me that … I wondered if … All of a sudden ... The feeling hit me like … I looked to see if … I noticed that … I looked again at …. I decided that … It occurred to me that … I imagined … I searched the scene for … Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Act Evaluate Wait Perceive Receptive Cognition Active Cognition The action-perception cycle is discussed in psychology, but little modeled in neuroscience. In the middle is anticipation (either conscious or not) leading to and shaping action selection. Anticipate React Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Cortical Macrodynamics Organized field activities are also coherent, spatially organized phenomena in the electrical ‘space’ of the cortex. Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Scott Makeig Event-Related Brain Dynamics I

Spatiotemporal dynamics Event-Related Brain Dynamics I are complex … Scott Makeig Event-Related Brain Dynamics I

Brain Structure & Dynamics Event-Related Brain Dynamics I MICRO Brain Structure & Dynamics Ephaptic Dynamics BEHAVIOR ECOG / EEG / MEG Recorded ?!≈ ? JUST DO IT MACRO RT Emergent Dynamics ~million GHz ~1 Hz BOLD Observable Scott Makeig Event-Related Brain Dynamics I

Brain Dynamics are Multiscale Event-Related Brain Dynamics I EEG (scalp surface) ECOG (cortical surface) Local Extracellular Fields Partial coherence in time and space of distributed field activity at each spatial scale produces the “signals” recorded at the next larger spatial scale. Intracellular fields and spikes Synaptic potentials Unmodeled portions of signals recorded at any spatial scale are often dismissed as irrelevant (“noise”) by researchers working at either larger or smaller scales… Scott Makeig Event-Related Brain Dynamics I

‘Spike-Wave Duality’ in Neuroscience Event-Related Brain Dynamics I Spikes and waves ‘Spike-Wave Duality’ in Neuroscience Field dynamics Waves Oscillations Chaos ? Spike dynamics Bursts Avalanches Electrotonic events Scott Makeig Event-Related Brain Dynamics I

‘Spike-Wave Duality’ in Neuroscience Event-Related Brain Dynamics I Field dynamics Waves Oscillations Chaos ? Spike dynamics Bursts Avalanches Electrotonic events Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Standard spike rate coding model: Quasi-thermal information conductance? “Hot” burst  diffuse “warmth” … Rate coding = neural info. transmission via intense stochastically- emitted bursts of spike activity (cf. “heat”). Bursts of spikes from one area  Sufficient synchrony to trigger spikes in target area(s). “Hot” burst in area A  “hot” burst in B  “hot” burst in C … = “quasi-thermal information conductance” But this is highly inefficient: More Energy Less Spatial resolution Less Temporal resolution Spike Burst Diffusivity Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Opposite Extreme: Spike Multiplexing Each spike train may participate in carrying more than one neural signal… i.e. Spike trains as multiplexed signals Each spike in the train may belong to a different, spatially distributed “volley” event and thus participate in transmitting a different neural “word”… Advantages: Efficient Flexible High spatial & temporal bandwidth Spike Train Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I What creates Synchronous Input Volleys ? Electrotonic coupling (threshold sculpting) Spike time dependent learning Neural-glial interactions Extracellular field biasing (ephaptic effects) Myelin growth control (conductance speed regulation) etc.etc. … Scott Makeig Event-Related Brain Dynamics I

Spike-Timing Dependent Learning Event-Related Brain Dynamics I Does spike synchrony have functions? Spike-Timing Dependent Learning Synchrony Rewarding / Promoting Scott Makeig Event-Related Brain Dynamics I Bi & Poo, 1998

Spike-Timing Dependent Learning Event-Related Brain Dynamics I Synchrony Rewarding / Promoting Scott Makeig Event-Related Brain Dynamics I Bi & Poo, 1998

Event-Related Brain Dynamics I Do fields have functions? No: Useless “roar of the crowd …” Yes, as indicator: Useful index of local synchrony Yes! They regulate synchrony (ephaptic effects) Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Ephaptic field effects Francis, Gluckman & Schiff (J Neurosci, 2003) applied external fields to a hippocampal slice and demonstrated local field effects on neural spiking down to well below the density of hippocampal LFP  nearly down to a predicted physical bound.  lowest field intensities produced stronger spike synchrony ! Scott Makeig Event-Related Brain Dynamics I

Single Scalp Electrode Event-Related Brain Dynamics I NEURAL SYNCHRONIES NEURAL NETWORKS Single Neuron Scott Makeig Event-Related Brain Dynamics I

It takes a neuropile to raise a spike volley “It takes a village to raise a child.” –Hillary Clinton It takes a neuropile to raise a spike volley To produce a spike requires a near-synchronous spike input volley & a near-threshold external environment & a near-threshold internal environment & … & it takes a neuropile to use a spike volley. Scott Makeig Event-Related Brain Dynamics I

Multiscale brain communication 1. Spike synchrony, producing extra-cellular fields, and biasing of spike synchrony by extracellular fields, must occur across different spatial scales, with different effects. 2. The spatial scales of partial synchrony giving rise to scalp-recorded fields are currently unknown, but might be extracted from (future) multiscale recordings. Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Brain Electrophysiology 1960 Response Averaging MEG ERF ERP  EEG  LFP  Spike Average Peri- Stimulus Histogram EEG  LFP The advent of response averaging help split the field of brain electrophysiology in two distant camps – those who average scalp potentials (ERPs) and those who study spike trains of single neurons. Currently, new and accelerating streams of results are linking ERP and EEG processes (e.g. using event-related coherence), on one hand, and spikes and local field potential oscillations (in awake behaving animals) on the other. A still-missing link is the connection between lthe <1-cm coherence domains of local cortical field potentials (LFPs) and the (?>1-cm?) EEG coherence domains. This gap (when completed by EEG/fMRI and surgery patient experiments) may at last unify the field of brain electrophysiology. Scott Makeig Event-Related Brain Dynamics I Makeig TINS 2002

Spatial Source Filtering Electrodes Cortex Local Synchrony EEG Skin Domains of synchrony EEG sources must be areas of partially synchronous local field potentials. Because of volume conduction, scalp electrodes (or MEG sensors) record the sum of potentials from different source areas. Spatial source filtering is therefore necessary to accurately measure the synchronized cortical activities. Local Synchrony Spatial Source Filtering Scalp sensors ‘mix’ the dynamics of cortical (and non-brain) sources Skull Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I MEG was acquired to see these processes in vivo L: Electrical current in apical dend  mag field R: sufficient number activ observable outside head. Scott Makeig Event-Related Brain Dynamics I R. Ramirez, 2005

Event-Related Brain Dynamics I Limitations of response averaging The response averaging model: EEG ERP EEG “noise” Data  Average + “Background” BOLD ERB BOLD “noise” But, this linear decomposition is veridical if & only if: 1. The Average appears in each trial. 2. The “Background” is not perturbed in other ways by the time locking events. Not True / Not Defined Not True Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I The response averaging model: EEG ERP EEG “noise” Data  Average + “Background” BOLD ERB BOLD “noise” But, this linear decomposition is veridical if & only if: 1. The Average appears in each trial. 2. The “Background” is not perturbed in other ways by the time locking events. Scott Makeig Event-Related Brain Dynamics I

The adequacy of blind response averaging IF …. If ‘equivalent’ stimuli (passively) evoke the same macro field responses (with fixed latencies and polarities or phase) in all trials… If all the REST of the EEG can be considered to be Gaussian noise sources that are not affected by the stimuli.. THEN … The stimulus-locked average contains all the meaningful event-related EEG/MEG brain dynamics. The adequacy of the ‘blind’ response averaging paradigm has been too little questioned in electrophysiology… Scott Makeig Event-Related Brain Dynamics I

EEGdata  ERPmean + EEGNOISEh The inadequacy of blind response averaging EEGdata  ERPmean + EEGNOISEh ? EEG1 EEG2 EEG4 BUT this simple model involves some highly questionable assumptions: ? The living brain produces passive responses ?? ? Ongoing EEG processes are not perturbed by events?? ? Evoked response processes are spatially segregated from ongoing EEG processes ?? ? ‘Equivalent’ stimulus events evoke equivalent brain responses  event-related brain dynamics are stationary from trial to trial ?? ? The ‘true’ response baseline is flat ?? Blind response averaging does not capture all the important features of event-related brain dynamics (!). Taking the results of response averaging as ‘real’ often leads to both explicit and tacit confusions… Scott Makeig Event-Related Brain Dynamics I

Monkey LOOK … Monkey Do Monkey see … Monkey Do Currently, much ERP research (as well as much neuroscience in general), remains focused on bottom-up (afferent) brain processing of stimulus events – As this figure (based on one published in Science, 2001) shows. However, anatomically, much or even most of the visual system is devoted to top-down (efferent) communication, whose role has until recently been all but ignored. Monkey do … Thorpe and Farbe-Thorpe, Science (2001) 291: 261 Scott Makeig Event-Related Brain Dynamics I

EEG? EEG? EEG? EEG? EEG? EEG? EEG? ERP ERP ERP ERP ERP C20-50 However, much ERP research (as well as much neuroscience in general), remains focused on bottom-up (afferent) brain processing of stimulus events – As this figure shows. However, anatomically, much or even most of the visual system is devoted to top-down (efferent) communication, whose role has until recently been all but ignored. Thorpe and Farbe-Thorpe, Science (2001) 291: 261 Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I A richer model Modeling Event-Related Brain Dynamics Un-mix cortical (and artifact) source contributions to the scalp electrodes using independent component analysis (ICA). Visualize the activities of independent component (IC) sources across single trials using ERP-image plotting. Model the event-related dynamics of the IC sources using time/frequency analysis. Localize the separated IC sources using inverse source mapping methods. Compare similarities in IC dynamics and locations across subjects using IC cluster analysis. Assess reliability of differences between IC activities time-locked to conditions, groups, and/or sessions of a study. Scott Makeig Event-Related Brain Dynamics I Photo: www.AlanBauer.com S. Makeig, 2006

Event-Related Brain Dynamics I Modeling Event-Related Brain Dynamics Un-mix cortical (and artifact) source contributions to the scalp electrodes using independent component analysis (ICA). Visualize the activities of independent component (IC) sources across single trials using ERP-image plotting. Model the event-related dynamics of the IC sources using time/frequency analysis. Localize the separated IC sources using inverse source mapping methods. Compare similarities in IC dynamics and locations across subjects using IC cluster analysis. Assess reliability of differences between IC activities time-locked to conditions, groups, and/or sessions of a study. Scott Makeig Event-Related Brain Dynamics I Photo: www.AlanBauer.com S. Makeig, 2006

Event-Related Brain Dynamics I Event-related perturbations ERP Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I ms 140 160 180 200 Potential (mV) -1 -2 1 15 Ss ERP +2.5 0 mV -2.5 400 600 800 Time (ms) Amplitude (dB) 31 Channels Scott Makeig Event-Related Brain Dynamics I Makeig et al., Science, 2002

Event-Related Brain Dynamics I 15 20 25 30 2 6 10 14 18 EEG Spectrum + mean -dB Frequency (Hz) Amplitude (dB) 31 Channels 2.5 6 10.5 16 Hz Rel. Power (dB) 2 10 14 18 Frequency (Hz) ERP Spectrum -10 500% 2 6 10 14 18 Freq. (Hz) 100 Amplitude (%) 300 500 Expected (1/√N) ERP/EEG Ratio 100% Scott Makeig Event-Related Brain Dynamics I Makeig et al., Science, 2002

Event-Related Brain Dynamics I 10-Hz Coh. Post. Cing. Fusiform Ant. Cing. A similar distributed coherence event was recently observed by Klopp et al. in intra-cortical data recorded pre-surgically from epileptic patients, also beginning 150 ms after presentation of a visual (face) stimulus. J Klopp, K Marinkovic, P Chauvel, V Nenov, E Halgren Hum Br Map 11:286-293 (2000) Scott Makeig Event-Related Brain Dynamics I

New Concepts  New Measures ERSP – event-related spectral power ITC – inter-trial coherence (phase locking) ERC – event-related coherence New Measures  New Visualizations erpimage() – sorted trial-by-trial dynamics envtopo() – ERPs and components tftopo() – event-related spectral power changes Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I ERP-Image Plotting Display single trials as color-coded horizontal lines (e.g., red is +µV, blue is -µV, green is 0). Sort all trials according to some variable of interest (here, subject RT). Smooth vertically. ERP-image plotting (Jung et al., 1999; Makeig et al., 1999) reveals trial-to-trial dependence on external or internal sorting variables (here, subject reaction time, RT). Here, on the left, ERP-image plotting shows the ‘P3’ feature of the visual target ERP (lower trace) at scalp site Cz is actually time locked to the subject button press, and is therefore not well represented in the stimulus-locked average. The right image shows the effects of smoothing the ERP image on the left using a (vertical) moving average of ~20 trials. Note the complex relationship of the ‘P2’ feature to subject RT … Scott Makeig Event-Related Brain Dynamics I Jung et al., Human Brain Mapping, 2001.

Event-Related Brain Dynamics I Collections of single trials are regular, but in multiple ways – so they appear noisy! Stim RT The ERP Image Detail of an ERP-image plot … Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I time EEG_epoch EEG_epoch EEG_epoch ERP RT EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch ERP image ERP-image plotting theory. Instead of averaging all trials (here conceived as ‘living’ in a multidimensional space of trial differences), creating one average ERP (red lines), A sorting variable (here, a sorting direction) is used to group trials along a one-dimensional continuum (straight orange arrow), producing an ERP image. However, many sorting directions are possible, even curving lines (curving orange arrow). Thus, there are many ERP-image visualizations of a set of event-related trials… ERP Cz One ERP Scott Makeig Event-Related Brain Dynamics I

Many ERP-image projections Event-Related Brain Dynamics I time EEG_epoch EEG_epoch EEG_epoch RT EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch ERP image ERP-image plotting theory. Instead of averaging all trials (here conceived as ‘living’ in a multidimensional space of trial differences), creating one average ERP (red lines), A sorting variable (here, a sorting direction) is used to group trials along a one-dimensional continuum (straight orange arrow), producing an ERP image. However, many sorting directions are possible, even curving lines (curving orange arrow). Thus, there are many ERP-image visualizations of a set of event-related trials… ERP Cz Many ERP-image projections Scott Makeig Event-Related Brain Dynamics I

Many ERP-image projections Event-Related Brain Dynamics I time EEG_epoch EEG_epoch EEG_epoch RT EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch EEG_epoch ERP image ERP-image plotting theory. Instead of averaging all trials (here conceived as ‘living’ in a multidimensional space of trial differences), creating one average ERP (red lines), A sorting variable (here, a sorting direction) is used to group trials along a one-dimensional continuum (straight orange arrow), producing an ERP image. However, many sorting directions are possible, even curving lines (curving orange arrow). Thus, there are many ERP-image visualizations of a set of event-related trials… ERP Cz Many ERP-image projections Scott Makeig Event-Related Brain Dynamics I

Blind EEG Source Separation  ICA Event-Related Brain Dynamics I Unmixes scalp channel mixing by volume conduction! CSF EEG Cocktail Party Independent Component Analysis (ICA) (right) can be used to separate N sound sources summed in recordings at N microphones, without relying on a detailed phonological model of the sounds characteristics of each source – this is so-called “blind separation.” ICA uses the presumption that the waveforms of the individual sound sources are independent over time. Applied to EEG data (left), ICA assumes that the EEG is predominantly composed of a number of domains of synchronous neural (or neuroglial) activity, each of which must, by simple biophysics, project to most of the recording scalp electrodes. If synchronous activity within these domains are predominantly independent of each other, ICA can separate the summed signals from these domains into records of their separation activities, given that the number of such domains making large contributions to the recorded signals are smaller than the number of recording sites. Scott Makeig Event-Related Brain Dynamics I

Blind EEG Source Separation  ICA Event-Related Brain Dynamics I Unmixes scalp channel mixing by volume conduction! CSF EEG Cocktail Party Independent Component Analysis (ICA) (right) can be used to separate N sound sources summed in recordings at N microphones, without relying on a detailed phonological model of the sounds characteristics of each source – this is so-called “blind separation.” ICA uses the presumption that the waveforms of the individual sound sources are independent over time. Applied to EEG data (left), ICA assumes that the EEG is predominantly composed of a number of domains of synchronous neural (or neuroglial) activity, each of which must, by simple biophysics, project to most of the recording scalp electrodes. If synchronous activity within these domains are predominantly independent of each other, ICA can separate the summed signals from these domains into records of their separation activities, given that the number of such domains making large contributions to the recorded signals are smaller than the number of recording sites. Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Independent Equivalent Dipole Scalp Distribution Domains of Synchrony Cortex Thalamus Scott Makeig Event-Related Brain Dynamics I

Event-Related Brain Dynamics I Sample EEG Decomposition Scott Makeig Event-Related Brain Dynamics I Onton & Makeig, 2006

Event-Related Brain Dynamics I A New Beginning… Scott Makeig Event-Related Brain Dynamics I