Modeling Large Scale Neural Systems Barry Horwitz Brain Imaging & Modeling Section National Institute on Deafness & Other Communication Disorders National Institutes of Health N1245

Collaborators Malle Tagamets Fatima Husain Theresa Long Brent Warner Julie Fitzer Jieun Kim Allen Braun Yukiko Kikuchi Mort Mishkin Feng Rong Jose Contreras-Vidal N1245

Methods to Understand Neural Basis of Human Cognition 1. Brain lesions & cognitive neuropsychology 2. Pharmacological and genetic studies 3. Electrophysiological recordings in primates (mammals) 4. Transcranial magnetic stimulation 5. Functional neuroimaging Hemodynamic-metabolic methods (PET, fMRI) Electric-magnetic methods (EEG, MEG) N1245 All these data are generally incommensurate with one another.

Methods to Understand Neural Basis of Human Cognition 1. Brain lesions & cognitive neuropsychology 2. Electrophysiological and lesion studies in primates (mammals) 3. Pharmacological and genetic studies 4. Transcranial magnetic stimulation 5. Functional neuroimaging Hemodynamic-metabolic methods (PET, fMRI) Electric-magnetic methods (EEG, MEG) N1245 Functional neuroimaging is unique – most of the other techniques permit one to examine only one neural entity at a time. Functional brain imaging allows one to study neural networks directly.

Problems with Relating Hemodynamic Data to Underlying Neural Activity 1. Spatial resolution - each PET or fMRI resolvable element contains multiple and diverse neuronal populations. 2. Temporal resolution - temporal resolution of neuronal activity is on the order of milliseconds; PET and fMRI (because of hemodynamic delay) is on the order of seconds; fast transients may be invisible to PET/fMRI. 3. Synaptic vs. neuronal activity - electrical activity comes from cell body firings, PET/fMRI reflect primarily the activity of synapses; excitatory vs. inhibitory. 4. Connectivity - PET/fMRI activity is a mixture of local and afferent synaptic activity. N1247

Large-Scale Neural Modeling Goal: Construct a large-scale, neurobiologically realistic neural model that can perform tasks like those studied by PET and fMRI. Multiple, interconnected brain regions (feedforward and feedback connections). Each region consists of multiple neuronal units (cortical column). The basic unit consists of an excitatory-inhibitory pair. Model can perform multiple tasks (e.g., DMS for shape, control task). Dynamic behavior of excitatory units in each region matches that observed by primate electrophysiological studies. Synaptic activity (both excitatory and inhibitory), integrated spatially and temporally, represents rCBF/BOLD. N1233

Uses for Large-Scale Modeling Understand how cognitive and sensorimotor processes are implemented neurally (forward modeling). Test experimental design and data analysis methods. Method to understand the neural substrate for high-level concepts. Method to combine multimodality information (e.g., fMRI, MEG, lesion). N1247

Perceptual Objects Perceptual Object Subject to figure-ground separation* Examples of visual objects Nameable object (e.g., dog, table) Delimited pattern Examples of auditory objects** Word Melodic fragment Definable & delimited environmental sound N1245 *Kubovy and Van Valkenburg, Cognition, 2001 ** Griffiths and Warren, Nat. Rev. Neurosci., 2004

From Signal to Percept to Concept (higher cortical levels) Percept (image) (primary cortex) N1245 Signal (receptors) Roy Patterson et al.

Neuroanatomy for Visual and Auditory Object Processing Kass et al.

Delayed Matched-to-Sample Tasks Stimulus 1 Stimulus 2 Delay Response ITI, next trial TIME Frequency Shape Tonal pattern

Cerebral Cortex 8: 310-320 (1998)

Regions of the Visual Model LGN (stimulus) N1245 V1/V2 Prefrontal V4 IT (Tagamets & Horwitz, Cerebral Cortex, 1998)

Basic Unit of Model and Between-Area Connections (Cortical column) 60% E 15% 15% 10% I 1. One excitatory (E) and one inhibitory (I) element per unit. 2. Local connections based on anatomical data. 3. Total afferent input ~ 10-15% local connections. 4. Sigmoidal activation rule.

PFC IT FS LGN D1 FR D2 Attention V1/V2 V4 Horizontal selective units Excitatory EE Inhibitory EI Horizontal selective units FS Corner selective units IT LGN D1 FR Vertical selective units Vertical selective units D2 Attention PFC V1/V2 V4

The Sigmoidal Activation Rule where å + = k I ki E iI t w in ) ( Multiparameter differential equation Δ = rate of increase d = rate of decay

Working Memory Module (IT-PF component) PF-s IT PF-d1 PF-r (and other areas) excitatory inhibitory ee ei PF-d2 s = cue-selective d1 = delay d2 = delay+cue r = response Modulator of Attention

Stimulus and Response N1245

Delay Period N1245

Response to Match N1245

Visual Model

Results and Conclusions: Visual Model Simulations with the Full Model (% CHANGE WITHIN AREAS) High Attention to Shape - Low Attention to Degraded Shape V1/V2 V4 IT Prefrontal +3.1% +5.2% +2.5% +3.5% Experimental Results (Haxby et al., 1995) +2.7% +8.1% +4.2% +4.1% Conclusions 1. Electrical activities in each region match exp. results in primates. 2. PET activities in each region match exp. results in humans. 3. Our hypothesis about how different frontal neuronal populations interact is supported. 4. Our hypothesis about relation between integrated synaptic activity and PET/fMRI data is supported. 5. Hypothesis about role of top-down processing is supported. N1238

NeuroImage 21: 1701-1720 (2004)

Regions of the Auditory Model (Husain et al., Neuroimage, 21: 1701-1720, 2004) Ai Aii STG/STS D1 D2 FR FS PFC MGN

Network Diagram of Auditory Model MGN Up Selective Units Down Contour STG/ STS FS Freq Time Freq Time Up Selective Units Freq Time Freq Time Up Selective Units FS FS FS Freq Time Freq Time Contour Selective Units STG/ STS STG/ STS STG/ STS D1 D1 D1 R R MGN Down Selective Units D2 D2 D2 Down Selective Units PFC Ai Attention Aii Excitatory E  E Inhibitory E  I FS = cue selective D1 = delay selective D2 = delay+cue selective R = response selective

Optical imaging - Chinchilla Simulated & Measured Activity in Auditory Regions: Increasing Temporal Windows of Integration Temporal window of increases along the Ai-Aii-STG/STS pathway. --- stimulus --- Ai --- Aii --- ST Optical imaging - Chinchilla (Harrison et al., 2000) Neural Firing Rate 10 Time- steps Time 0.5 1.0 Input Ai Aii ST Timesteps 100 timesteps Intensity

Monkey call-related activity wn 40 5 -500 500 1000 10 20 30 40 6 -500 500 1000 10 20 30 40 1 -500 500 1000 10 20 30 40 2 3 -500 500 1000 10 20 30 40 4 30 20 10 -500 500 1000 -500 500 1000 10 20 30 40 7 12 -500 500 1000 10 20 30 40 13 8 -500 500 1000 10 20 30 40 9 10 -500 500 1000 20 30 40 11 -500 500 1000 10 20 30 40 20 -500 500 1000 10 30 40 14 -500 500 1000 10 20 30 40 15 16 -500 500 1000 10 20 30 40 17 -500 500 1000 10 20 30 40 18 -500 500 1000 10 20 30 40 19 -500 500 1000 10 20 30 40 MC 27 -500 500 1000 10 20 30 40 21 -500 500 1000 10 20 30 40 22 -500 500 1000 10 20 30 40 23 -500 500 1000 10 20 30 40 24 25 -500 500 1000 10 20 30 40 26 -500 500 1000 10 20 30 40 10 20 30 40 34 -500 500 1000 28 -500 500 1000 10 20 30 40 29 -500 500 1000 10 20 30 40 31 32 -500 500 1000 10 20 30 40 33 others complex 40 -500 500 1000 10 20 30 41 35 -500 500 1000 10 20 30 40 36 -500 500 1000 10 20 30 40 37 38 -500 500 1000 10 20 30 40 39 FM 42 -500 500 1000 10 20 30 40 43 44 PT P < 0.01 XD122303-1S SPK ch# 4 ID#9

Simulation of fMRI Experiments fMRI activity is simulated by spatial and temporal integration of the absolute value of the synaptic activity over 50 msec (which represents the time needed to acquire an fMRI slice). This time course is then convolved with a Poisson function representing the hemodynamic delay. The resulting function is then sampled every Tr sec (volume acquisition time) to yield the simulated fMRI activity during each scan series. (Horwitz and Tagamets, Human Brain Mapp., 1999)

BLOCK design paradigm; task blocks interspersed with rest. Experimental Details Timeline of a Trial Stimulus 1 0.35 sec Delay 1.0 sec Stimulus 2 0.35 sec Response 2.0 sec Next Trial Task: Subjects discriminate between the two sounds they hear based on whether the two sounds are exactly the same. BLOCK design paradigm; task blocks interspersed with rest. Subjects: 12 normal, right-handed, American English speakers (5 women, 7 men). Training: ½ hour before scanning. Subjects performed at 85% correct. Behavioral data collected via button presses. 22 axial slices, collected on a 1.5 Tesla GE scanner. Analysis conducted using the statistical parametric mapping (SPM99) software. Preprocessing, fixed-effects analysis

Auditory Stimuli vs. Rest Tones R L Tonal Contours

Regions of Interest for Auditory Model z=15 z= 9 Ai Aii (red), PFC(green) ST z= 3 Left Ai Aii ST PFC z = 9 z = 3 z=6 Right

Model vs. Experiment: % Signal Change (Tonal Contours-Tones) 100.0 80.0 Expt_right Percentage Signal Change 60.0 Model 40.0 20.0 0.0 Ai Aii ST PFC

Model vs. Experiment: % Signal Change (Tonal Contours-Tones) 100.0 80.0 Expt_right Percentage Signal Change 60.0 Model 40.0 20.0 0.0 Ai Aii ST PFC

Network Diagram of Auditory Model MGN Up Selective Units Down Contour STG/ STS FS Freq Time Freq Time Up Selective Units Freq Time Freq Time Up Selective Units FS FS FS Freq Time Freq Time Contour Selective Units STG/ STS STG/ STS STG/ STS D1 D1 D1 R R MGN Down Selective Units D2 D2 D2 Down Selective Units PFC Ai Attention Aii Excitatory E  E Inhibitory E  I FS = cue selective D1 = delay selective D2 = delay+cue selective R = response selective

Contour-selective neuron Brief Explanation: Contour-selective neuron Contour-selective neurons are selective to changes in sweep direction occurring within their time window. The duration of this sound is 300 ms. This neuron increased in activity as soon as the FM sweep changed the direction of its frequency. However, if you take a look at the activities of the same neuron to the stimuli with the same components, you can see that this neuron didn’t respond to the mere FM down or up sweep, or the pure tone at the same frequency. To elicit the activity of this neuron, the change of the direction of sweep is necessary. We cannot explain much about this neuron with the summation of neurons which we can observe in A1. This type of neuron might play a role in the linkage between the neurons which represent the tonotopic map and the neurons which engage in the higher auditory representations, such as species-specific vocalization processing.

General Conclusions We have constructed a large-scale, neurobiologically realistic model of cortical processing of auditory objects. Simulated neuronal activities agree with experimental findings. Simulated fMRI data agree with experimental findings. Model can account for a number of perceptual grouping observations (human psychophysical performance). Model predicted existence of a type of neuron found in primate cortex. Simulated MEG looks like it will agree with experimental MEG data.