1. Adapted from Hayrettin Gürkök, U. of Twente, NL
BMI Principles
Jose C. Principe
University of Florida
Adapted from Hayrettin Gürkök, U. of Twente, NL

2. Literature

3. Difficulties in Invasive BMIs
BCIs offer an easy entry to research
Non invasiveness straight forward data collection
Closer to cognition
Conventional signal processing
BMIs research infrastructure is much harder
Work with animals (ethics)
Difficult instrumentation
Unclear signal processing

4. Choice of Scale for Neuroprosthetics
Bandwidth (approximate)
Localization
Scalp Electrodes
0 ~ 80 Hz
Cortical Surface
Volume Conduction
3-5 cm
Electro-corticogram (ECoG)
0 ~ 500Hz
0.5-1 cm
Micro Electrodes
500 ~ 7kHz
Local Fields
1mm
Single Neuron
200 mm

5. Electrode Arrays Utah array Brain Gate Michigan probes
J. C. Sanchez, N. Alba, T. Nishida, C. Batich, and P. R. Carney, "Structural modifications in chronic microwire electrodes for cortical neuroprosthetics: a case study," IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2006

6. Technical Issues with BMIs
An implantable BMI requires beyond of state of the art technology:
Ultra low power
Ultra miniaturized
Huge data bandwidth/power form factor
Packaging

7. FWIRE: Florida Wireless Implantable Recording Electrodes
Thru vias to
RX/Power Coil +
12.5 mm
Coil winding
3.5 mm
50µm pitch
Electrodes
Coin Battery
(10 x 2.5 mm)
Battery
Supporting
screws
Flexible
substrate
TX antenna
Modular
Electrode
attachment
sites
IF-IC
RFIC
18 mm
28mm
15mm
12mm
Coil
Battery
Patterned
Substrate
Supporting
Electrode
Array IC
Flip-chip
connection
Specifications:
16 flexible microelectrodes (40 dB, 20 KHz)
Wireless (500 Kpulse/sec)
2mW of power (72-96 hours between charges)

8. RatPack Low-Power, Wireless, Portable BMIs
Requirements
Total Weight: < 100g
Battery Powered: Run for 4 hours
Implantable
Biocompatible
Heat flux: < 50 mW/cm2
Power dissipation should not exceed a few hundred milliwatts
Backpack
Small form factor
Speed vs. Low Power

9. UF PICO System
PICO system = DSP + Wireless
Generation 3

10. General Architecture
BCI (BMI) bypasses the brain’s normal pathways of peripheral nerves (and muscles)
J.R. Wolpaw et al. 2002

11. BMIs: How to put it together?
NeoCortical Brain Areas Related to Movement
Posterior Parietal (PP) – Visual to motor transformation
Premotor (PM) and Dorsal Premotor (PMD) -
Planning and guidance (visual inputs)
Primary Motor (M1) – Initiates muscle contraction
First, both are made from different materials.
Time and Scale – (neurons vs transistors) With this comparison, the time scales are about 1 ms for the neuron vs. 1 ns for the transistor.
The spatial scale is about 1 mm for the largest neuron vs. less than 1 μm for a modern CMOS transistor.
Serial - calculating step by step in a cookbook fashion from the beginning to the end of the calculation.

12. Motor Tasks Performed Data 2 Owl monkeys – Belle, Carmen
2 Rhesus monkeys – Aurora, Ivy
sorted cells
Cortices sampled: PP, M1, PMd, S1, SMA
Neuronal rate (100 Hz) and behavior is time synchronized and downsampled to 10Hz
Task 2

13. 100 msec Binned Counts Raster of 105 neurons (spike sorted)

14. Ensemble Correlations – Local in Time – are Averaged with Global Models

15. Computational Models of Neural Intent
Three different levels of neurophysiology realism
Black Box models – function relation between input - desired response (no realism!)
Generative Models –state space models using neuroscience
elements (minimal realism).
White models – significant realism (wish list!)

16. Optimal Linear Model The Wiener (regression) solution
Normalized LMS with weight decay is a simple starting point.
Four multiplies, one divide and two adds per weight update
Ten tap embedding with 105 neurons
For 1-D topology contains 1,050 parameters (3,150) w0 w9
Z-1 delay of 1 sample
S adder
wi(n) parameter i at time n

17. 3-D, 2-D Trajectory Modeling and Robot Control
Collaboration with Miguel Nicolelis, Duke University
Sponsored by DARPA

18. Time-Delay Neural Network (TDNN)
The first layer is a bank of linear filters followed by a nonlinearity.
The number of delays to span I second
y(n)= Σ wf(Σwx(n))
Trained with backpropagation
Topology contains a ten tap embedding and five hidden PEs– 5,255 weights (1-D)
Principe, UF

19. Multiple Switching Local Models
Multiple adaptive filters that compete to win the modeling of a signal segment.
Structure is trained all together with normalized LMS/weight decay
Needs to be adapted for input-output modeling.
We selected 10 FIR experts of order 10 (105 input channels)
d(n)

20. Recurrent Multilayer Perceptron (RMLP) – Nonlinear “Black Box”
Spatially recurrent dynamical systems
Memory is created by feeding back the states of the hidden PEs.
Feedback allows for continuous representations on multiple timescales.
If unfolded into a TDNN it can be shown to be a universal mapper in Rn
Trained with backpropagation through time

21. Generative Models for BMIs
Use partial information about the physiological system, normally in the form of states.
They can be either applied to binned data or to spike trains directly.
Here we will only cover the spike train implementations.
Difficulty of spike train Analysis:
Spike trains are point processes, i.e. all the information is contained in the timing of events, not in the amplitude of the signals!

22. Particle Filters for Point Processes
Linear filter
nonlinearity f
Poisson model
kinematics
spikes
Instantaneous tuning model
Kinematic State
Neural Tuning function
spike trains
Prediction
Updating
NonGaussian
P(state|observation) 22

23. Generative Data Modeling
…..
…..
Hidden
Processes
(Brain areas)
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
…..
Neural Channels
…..
…..
…..
…..
Observable
Processes
(probed neurons)
…..
…..
…..
Time

24. BMI lessons learned BMIs are beyond the Proof of Concept stage, but….
Present systems are signal translators and will not be the blue print for clinical applications
Current decoding methods use kinematic training signals - not available in the paralyzed
I/O models cannot contend with new environments without retraining
BMIs should not be simply a passive decoder – incorporate cognitive abilities of the user
What are we going to do with this information? Bring together a novel BMI architecture. 24

25. BMI lessons learned BMIs are beyond the Proof of Concept stage, but….
Present systems are signal translators and will not be the blue print for clinical applications
Current decoding methods use kinematic training signals - not available in the paralyzed
I/O models cannot contend with new environments without retraining
BMIs should not be simply a passive decoder – incorporate cognitive abilities of the user
What are we going to do with this information? Bring together a novel BMI architecture. 25

26. A Paradigm Shift for BMIs!
Neural Signal Processing
DSP algorithm
Desired response
During training the user actions create a desired response to the DSP algorithm.
During testing the DSP algorithm creates an approximation to the desired response. 26

27. Neural Signal Processing
A Paradigm Shift for BMIs!
Neural Signal Processing
Control Algorithm
Learning Algorithm X
The control algorithm learns through reinforcement to achieve common goals in the environment.
Shared control with user to enhance learning in multiple scenarios and acquire the net benefits of behavioral, computational, and physiological strategies 27

28. Construction of a New Framework How to capitalize on the perception-action cycle?
The brain is embodied and the body is embedded
Need to quantify Brain State at different time resolutions
Intelligent behavior arises from the actions of an individual seeking to maximize received reward in a complex and changing world.
The BMI must engage and dialogue with the user:
Exploits better engineering knowledge
Utilizes cognitive states
Dissects behavior top-down
Exploits rewards
Learns with use
Propose Reinforcement Learning to train the BMI.
FUTURE
PAST
INTERNAL
REPRESENTATION
EXTERNAL
WORLD
LIMBIC
SYSTEM
ORGANIZED PAST
EXPERIENCE
PREDICTIVE
MODELING
DOES ACTION
MEET
FUTURE REALITY?
SENSORY
STIMULUS
Causality line
Body line 28

29. Reward Learning Involves a Dialogue
Relation between the agent and its environment.
Environment: You are in state 14. You have 2 possible actions.
Agent: I'll take action 2.
Environment: You received a reinforcement of 17.8 units. You are now in state 13. You have 2 possible actions.
Agent: I'll take action 1.
repeat
AGENT
ENVIRONMENT
actions
rewards
states
Examples:
1) Riding a bike: Silvestro will not tell Francisco to follow a specific temporal sequence and phase of knee and ankle flexion angles while maintaining a specific center of gravity.
2) Learning to cook sauce: [SL] follow an exact recipe – fastest but inflexible. [RL] very slow, trial and error.
SEGUE: How can this system be implemented?
Goal
Start 29

30. CABMI involves TWO intelligent agents in a cooperative dialogue!!!
User’s neuromodulation
sets the value function for the CA
COMPUTER AGENT
actions
rewards
states
environment
ROBOT
Active assistant to the paralyzed patient.
Interaction is not passive but grows through experience.
Both the CA and the user have the same reward in 3D space
RAT’S BRAIN
RAT’S BRAIN 30

31. Features of co-adaptive BMI
Enables intelligent system design in BMIs
Both systems adapt in close loop in a very tight coupling between brain activity and computer agent ( CA states are specified by brain activity).
User must incorporate the CA in its world (can a rat learn this?)
CA must decode brain activity for its value function (can it model the signature of behavior?).
In fact CABMI is a “symbiotic” biological-computer hybrid system. 31

32. Experiment workspace [top view]
The user learns first to associate
levers with water reward in a training phase.
In brain control, it progressively associates the blue guide LED of the robotic arm with the target lever LEDs.
Only when the robot presses the target lever it will get reward.
We developed an animal model of a BMI. [robot arm lengths 25 – cm] 32

33. Experiment workspace [top view]
We developed an animal model of a BMI. [robot arm lengths 25 – cm] 33

34. Experimental CABMI Paradigm
Grid-space
Incorrect
Target
Correct
Starting
Position
Map workspace to grid
Robot Arm
Rat
Match LEDs
27 discrete actions
26 movements
1 stationary
Match LEDs
Rat’s Perspective
Left Lever
Water Reward
Right Lever

35. Experimental CABMI Paradigm
CA rewards are defined in 3D at the target lever positions.
RL is used to train the CA in brain control (tabula rasa, i.e. no a priori information).
During brain control, shaping of the reward field increases the level of difficulty across multiple sections with an adjustable threshold target. 35

36. Neuromodulation defines the States
Sampling rate 24.4 kHz
Hall, Brain Research (1974)
In the next five slides I will cover ~28 years of invasive, single unit BMI, so bear with me.
Bilateral Premotor/motor
Areas
32 channels
Spike sorted data 36

37. Performance metrics Performance metrics:
Percentage of trials earning reward
Average control time required to reach a target
4 sessions were simulated using random action selection to estimate chance performance for the CABMI in increasing difficulty tasks.
Chance will always be our performance comparison
Chance PR is calculated using five sets of 10,000 simulated brain control trials using random action selection. The PR from each set of trials is then used to calculate the average and standard deviation. Chance TT is calculated from the concatenation of the 5 sets of random trials. The data used to calculate chance PR and TT is also used in K-S tests for statistical comparisons. 37

38. % trials earning reward time to achieve reward
Performance in 4 tasks of increasing difficulty
% trials earning reward time to achieve reward
460%, 517%, and 515% 38

39. Closed-Loop RLBMI Robot workspace in rat visual field of view.
BLUE – Robot
GREEN - Lever
Functional levers
Non-functional levers
Top-view of the rat behavioral cage.

40. Event Related Desynchronization (ERD) and synchronization (ERS)
It is well established that preparation, execution, and also imagination of movement produce an event-related desynchronization (ERD) over the sensorimotor areas, with maxima in the alpha band (mu rhythm, 10 Hz) and beta band (20 Hz).
The mu ERD is most prominent over the contralateral sensorimotor areas during motor preparation and extends bilaterally with movement initiation
ERD during hand motor imagery is very similar to the pre-movement ERD, i.e., it is locally restricted to the contralateral sensorimotor areas

41. Event Related Desynchronization (ERD) and synchronization (ERS)
During movement preparation and execution, an increase of synchronization (ERS) in the 10-Hz band normally appears over areas not engaged in the task (idling)
ERS can also be observed after the movement, over the same areas that had displayed ERD earlier

42. Beta rebound following movement and somatosensory stimulation
The general finding is that beta oscillations are desynchronized during preparation, execution, and imagination of a motor act
After movement offset, the beta band activity recovers very fast (<1 s) and short-lasting beta bursts appear.
The occurrence of a beta rebound related to mental motor imagery implies that this activity does not necessarily depend on motor cortex output.
A number of experiments have also shown beta oscillations to be sensitive to somatosensory stimulation

43. ERS (Blue) and ERD (Red)
12.0 Hz +/- 1.0
10.9 Hz +/- 0.9
Pfurtscheller

44. ERS (Blue) and ERD (Red)
Pfurtscheller

45. Beta ERS
Pfurtscheller

46. Alpha and Beta ERS
Pfurtscheller

47. Signal Processing for ERD/ERS
Bandpass filtering between 9-13 Hz will emphasize this component.
Estimate the power
Place a statistical threshold for detection.
Alternatively use PSD and threshold the appropriate frequency band.

48. Paradigm 1 (

49. Paradigm 2

50. Event Related Potentials
ERPs are a signature of cognition. They signal a massive communication amongst brain areas (kind of the brain’s impulse response to an internal stimulus).
This is very good, but the problem is that it is normally much smaller than the ongoing EEG activity (i.e. the SNR is negative).

51. Event Related Potentials
The ERP shape is well known and pretty stable across individuals, and has a known distribution across the channels.
The P300 is the most used for BMIs because it is task relevant
N100-P200 complex is pre-attentive response
appearing over sensory areas
P300 signals a rare tasks relevant event (Cz)
N400 signals an unexpected event (Cz)

52. Event Related Potentials
In order to deal with the negative SNR, we use averaging of the stimulus.
If you have a transient that appears in white Gaussian noise, align the transient and average across trielas you obtain an increase of SNR by , where N is the number of trials.
This is normally done but has three shortcomings:
It is not real time
It assumes that the shape of the ERP is the same
It assumes that the latency is constant

53. P300 Event Related Potentials

54. P300 Event Related Potentials
Negative SNR so need averaging (i.e. repeated presentation of stimuli)

55. P300 Paradigm

56. P300 Paradigm

57. P300 Paradigm 2

58. The Cortical Mouse YES NO
In 1990 the CNEL proposed a new computer interface that would control cursor movement in the screen using directly brain activity (EEG) and implemented in a NeXT Computer
YES NO
Left/Right
4.5bits/min
Decision based on single ERPs (N400) in real time
Neural network classifier implemented in DSP chip
Overall control (synch, screen, data flow) by the OS
Konger, C., Principe, J., ANN classification of ERPs for a new computer interface IEEE IJCNN, 1990
Sina Eatemadi, A new computer interface using event related potentials University of Florida, 1992.

59. Slow Cortical Potentials

60. SCP Paradigm

61. Steady State Evoked Potential (ssEP)

62. ssVEP Paradigms

63. ssVEP Paradigms

64. ssVEP Paradigms One of the most reliable effects.
Need to do FFT of occipital channels and pick the highest frequency.
Car race (winner of the first BCI competition)

65. Taxonomy of BCI paradigms

66. Taxonomy of BCI paradigms

67. Taxonomy of BCI paradigms

68. Taxonomy of BCI paradigms

69. Mu Rhythm
When a subject imagines movement or sees movement made by others a burst of activity in the 8-12Hz range appears over the sensorimotor areas in the brain
The subject can synchronize the rhythm and by moving desynchronize it, hence it ia good signal to be used for motor BMI tasks.