Neuronal Ensembles in Motor Cortex Encoding & Decoding of Neuronal Ensembles in Motor Cortex Nicholas Hatsopoulos Dept. of Organismal Biology & Anatomy Committees on Computational Neuroscience & Neurobiology University of Chicago Co-founder & board member of Cyberkinetics, Inc.

Encoding Problem Decoding Problem Multi-trial averaging Behavior Single-trial prediction

“The motor cortex appears to be par excellence a synthetic organ for motor acts… the motor cortex seems to possess, or to be in touch with, the small localized movements as separable units, and to supply great numbers of connecting processes between these, so as to associate them together in extremely varied combinations. The acquirement of skilled movements, though certainly a process involving far wider areas (cf. V. Monakow) of the cortex than the excitable zone itself, may be presumed to find in the motor cortex an organ whose synthetic properties are part of the physiological basis that renders that acquirement possible.” Leyton & Sherrington (1917)

The two components of language Words or elementary primitives of meaning Rules or grammar by which the primitives are combined Pinker (1999)

The language of motor action in motor cortex Motor primitives: position, velocity, direction, trajectory Motor grammar: addition

Center-Out Task

Directional Tuning 150 frequency (Hz) NEXT SLIDE: Outline -0.5 1.0 9 ° 1 3 5 ° 4 5 ° 180° ° 2 4 2 2 5 ° 3 1 5 ° In this example, the neuron fires maximally for movements to the left and forward (i.e. 135 degrees) and minimally in the opposite direction. We could fit the tuning curve to a cosine function whose maximum value occurs at the so-called preferred direction of the cell. 80% of neurons we recorded from were well fit with a cosine function. NEXT SLIDE: Outline 2 7 ° 150 frequency (Hz) -0.5 1.0 time (s)

Behavioral Apparatus

Random-walk task A B

Chronic Multi-electrode array We have been using the Utah Intracortical electrode array developed by Dick Normann at the University of Utah. The array is silicon-based and is composed of 100 micro-electrodes arranged in a 10 by 10 matrix, each of which is separated from each other by 400 microns. So far, we have performed 24 implants in 12 animals recording from primary,dorsal premotor, and supplementary motor cortices. And we recorded from up to 50 neurons simultaneously. The figure shows an array implanted in the arm area of MI. Next Slide: Waveforms Utah/Bionic Technologies Probe Richard Normann U Utah

Primary Motor Cortex (MI) Leg Arm MI We have been using the Utah Intracortical electrode array developed by Dick Normann at the University of Utah. The array is silicon-based and is composed of 100 micro-electrodes arranged in a 10 by 10 matrix, each of which is separated from each other by 400 microns. So far, we have performed 24 implants in 12 animals recording from primary,dorsal premotor, and supplementary motor cortices. And we recorded from up to 50 neurons simultaneously. The figure shows an array implanted in the arm area of MI. Next Slide: Waveforms Face 5 mm

Long-term Reliability & Stability 0.8 0.7 0.6 0.5 ARRAY YIELD 0.4 0.3 0.2 subject 1 (tom) subject 2 (coco) 0.1 subject 3 (buddy) subject 4 (radley) 100 200 300 400 500 600 700 800 900 1000 Many Neurons Every Day (19 tests over 110 days) Blue - no recording Red - best recordings DAYS POSTIMPLANT

Why view motor cortical encoding as time-dependent? Trajectory-selective activity in motor cortex (Hocherman & Wise, 1990, 1991) Preferred directions shift in time (Mason et al., 1998; Sergio et al., 2005; Sergio & Kalaska, 1998)

Center-out task Shifts versus time

Center-out task Random-walk task Shifts versus time Shifts versus lead/lag time lag lead movement onset

Temporal tuning (information theory) Lead/lag (s)

The Encoding Model A class of general linear models (e.g. logistic regression) that estimates the probability of a spike given a particular movement trajectory: = preferred velocity trajectory integrated = preferred trajectory and path (“pathlet”)

model input = probability of a spike model input lead/lag (ms)

Temporal stability of pathlet representation

ROC analysis to quantify the performance of the encoding model

Encoding Performance as a function of trajectory length

Decoding Performance as a function of trajectory length

Map of Pathlets

Horizontal connectivity in motor cortex 1 mm Huntley & Jones (1991)

Rule for combining pathlets: Additive rule Assuming conditional independence,

Potential violations of conditional independence Case #1 Case #2 neuron 39 vs. neuron 51 -0.4 -0.2 0.2 0.4 20 40 60 80 100 neuron 40 vs. neuron 41 -0.4 -0.2 0.2 0.4 20 60 100 140 180 220 260 Counts/bin 1 ms bin

Spike Jitter Method (reference) (target) neuron 1 neuron 2 2 parameters: +/-w w, time resolution of synchrony J, the jitter window neuron 1 (reference) neuron 2 (target) +J -J 2 spike jittered trains This method evolved from a thought experiment that Elie Bienenstock had. Imagine … Would this magical drug destroy the apparent synchrony that we observe and if it did, what that disrupt thinking in the animal.

10% of all cell pairs (N=1431) show significant synchrony at a Case #1 Case #2 1000 jitters 10% of all cell pairs (N=1431) show significant synchrony at a resolution of +/-5 ms, p<0.05

Potential violations of conditional independence Case #1 Case #2

When conditional independence appears to be violated Conclusion: Synchronization preserves additive rule but increases the sensitivity of the tuning function by increasing the gain

Neuro-motor prosthetic system 2) Decoding of neural signals 1) Multi-electrode array implant Computer 3) Output interface I want to talk to you today about our efforts at Brown University with John Donoghue to try to develop a neuro-motor prosthetic system. The problem of using cortical signals to drive an external device can be broken up into three components. First, an chronic electrode implant is required. In our case, we have used a multi-electrode array to record from large numbers of single units. Second, these signals need to be decoded or mapped to a set of movement parameters. These parameters could include discrete behavioral states or continuous parameters such as position of the hand in 2 or 3 D space. Finally, these parameters are sent to a variety of output devices such a computer’s mouse or keyboard, or some sort of assistive robot, or even interface with biological tissue such as the muscles, peripheral nerves, or spinal chord gray matter. Personal computer: • mouse • keyboard Assistive Robotics: • robotic arm • mechanized prosthetic arm Biological interface: • muscles • peripheral nerve • spinal cord

BrainGateTM Pilot Device Cable Cart Sensor

BrainGateTM Sensor Implantation and Post-Op Recovery as Planned Surgery as planned Post-op recovery unremarkable Wound healing around pedestal complete Array on Cortex 2 months post implant Insertion

Binary Modulation-Imagined Opening/Closing of Hand

BrainGate Signal Detection and Analysis; SCI Able to Modulate Neural Output This slide shows the ability of patient #1 to modulate his neural output (from the motor cortex). When the operator says “left” you can immediately see activity in the top row of the display. This corresponds to one neuron. This neuron shows no activity in response to the “right” command.

( ) 2. Algorithmic level: Optimal Linear Filter Reconstruction Rf X(t) Response of neural ensemble in time Estimated position of hand in time Rf X(t) = ˆ filter coefficients ( ) X R f T 1 - = Essentially, this was a linear regression approach where we estimated the hand position, X, based on the product of the ensemble firing rates over time multiplied by a set of filter coefficients. This allowed us to come up with a least squares estimate for the filter coefficients. We then tested the regression model on data that had not been used to build the model. NEXT SILDE: RESULTS OF RECONSTRUCTION Warland et al. (1997)

Two Dimensional Cursor control

Hatsopoulos Lab Qingqing Xu Wei Wu, PhD Sunday Francis Zach Haga Jignesh Joshi John O’Leary Dawn Paulsen Jake Reimer Jonathan Ko Joana Pellerano Richard Penn, MD Matt Fellows Yali Amit Cyberkinetics, Inc.

Cross-Validated Performance as a function of pathlet length

Cross-validated Pathlet Population Vector Decoding