1. Encoding & Decoding of Neuronal Ensembles in Motor Cortex Nicholas Hatsopoulos Dept. of Organismal Biology & Anatomy Committees on Computational Neuroscience & Neurobiology University of Chicago Co-founder & board member of Cyberkinetics, Inc.

2. Encoding Problem trial 1 trial 2 trial 3 trial 4 trial 5 Decoding Problem Behavior Multi-trial averaging Single-trial prediction

3. “The motor cortex appears to be par excellence a synthetic organ for motor acts… the motor cortex seems to possess, or to be in touch with, the small localized movements as separable units, and to supply great numbers of connecting processes between these, so as to associate them together in extremely varied combinations. The acquirement of skilled movements, though certainly a process involving far wider areas (cf. V. Monakow) of the cortex than the excitable zone itself, may be presumed to find in the motor cortex an organ whose synthetic properties are part of the physiological basis that renders that acquirement possible.” Leyton & Sherrington (1917)

4. The two components of language Words or elementary primitives of meaning Rules or grammar by which the primitives are combined Pinker (1999)

5. The language of motor action in motor cortex Motor primitives: position, velocity, direction, trajectory Motor grammar: addition

6. Center-Out Task

7. Directional Tuning time (s) 0° 45° 90° 135° 180° 225° 270° 315° 2040 frequency (Hz) 150 0 -0.5 1. 0 0

8. Behavioral Apparatus

9. Random-walk task A B

10. Utah/Bionic Technologies Probe Richard Normann U Utah Chronic Multi-electrode array

11. Primary Motor Cortex (MI)Leg Face Arm 5 mm MI

13. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 01002003004005006007008009001000 DAYS POSTIMPLANT ARRAY YIELD subject 1 (tom) subject 2 (coco) subject 3 (buddy) subject 4 (radley) Long-term Reliability & Stability Many Neurons Every Day (19 tests over 110 days) Blue - no recording Red - best recordings

14. Why view motor cortical encoding as time-dependent? Trajectory-selective activity in motor cortex (Hocherman & Wise, 1990, 1991) Preferred directions shift in time (Mason et al., 1998; Sergio et al., 2005; Sergio & Kalaska, 1998)

15. Center-out task Shifts versus time

16. Center-out taskRandom-walk task Shifts versus timeShifts versus lead/lag time movement onset laglead

18. Temporal tuning (information theory) Lead/lag (s)

19. The Encoding Model A class of general linear models (e.g. logistic regression) that estimates the probability of a spike given a particular movement trajectory: = preferred velocity trajectory = preferred trajectory and path (“pathlet”) integrated

21. model input = model input probability of a spike lead/lag (ms)

22. Temporal stability of pathlet representation

23. ROC analysis to quantify the performance of the encoding model

24. Encoding Performance as a function of trajectory length

25. Decoding Performance as a function of trajectory length

26. Map of Pathlets

28. Horizontal connectivity in motor cortex 1 mm Huntley & Jones (1991)

29. Rule for combining pathlets: Additive rule Assuming conditional independence,

33. neuron 39 vs. neuron 51 -0.4-0.200.20.4 20 40 60 80 100 neuron 40 vs. neuron 41 -0.4-0.200.20.4 20 60 100 140 180 220 260 Counts/bin 1 ms bin Case #1Case #2 Potential violations of conditional independence

34. Spike Jitter Method neuron 1 ( reference ) +J -J +/-w 2 parameters: w, time resolution of synchrony J, the jitter window neuron 2 ( target ) 2 spike jittered trains

35. 10% of all cell pairs (N=1431) show significant synchrony at a resolution of +/-5 ms, p<0.05 Case #1 Case #2 1000 jitters

36. Potential violations of conditional independence Case #1 Case #2

38. Conclusion: Synchronization preserves additive rule but increases the sensitivity of the tuning function by increasing the gain When conditional independence appears to be violated

39. Computer Neuro-motor prosthetic system 1) Multi-electrode array implant 2) Decoding of neural signals3) Output interface Personal computer: mouse keyboard Assistive Robotics: robotic arm mechanized prosthetic arm Biological interface: muscles peripheral nerve spinal cord

40. Sensor Cable Cart BrainGate TM Pilot Device

41. BrainGate TM Sensor Implantation and Post-Op Recovery as Planned Surgery as planned Post-op recovery unremarkable Wound healing around pedestal complete Array on Cortex Insertion 2 months post implant

42. Binary Modulation-Imagined Opening/Closing of Hand

43. BrainGate Signal Detection and Analysis; SCI Able to Modulate Neural Output

44.  XRRRf T 1 T   Warland et al. (1997) 2. Algorithmic level: Optimal Linear Filter Reconstruction RfX(t)  ˆ Estimated position of hand in time Response of neural ensemble in time filter coefficients

45. Two Dimensional Cursor control

47. Hatsopoulos Lab Qingqing Xu Wei Wu, PhD Sunday Francis Zach Haga Jignesh Joshi John O’Leary Dawn Paulsen Jake Reimer Jonathan Ko Joana Pellerano Richard Penn, MD Matt Fellows Yali Amit Cyberkinetics, Inc.

48. Cross-Validated Performance as a function of pathlet length

49. Cross-validated Pathlet Population Vector Decoding