1. Non-Explicit Communication with Robotic Manipulators: A Review
Paul M. Calhoun

2. What Is It?
Communication with robots that is both explicit and direct is well defined
Joystics, words, gestures, are all both explicit and direct
Communication which is non-explicit, and non-direct is also well defined
The robot sees that an event has occurred or an object is in a location and takes action based on that
Keep it VERY short

3. What Is It? Non-Explicit but direct communication is rare
We want the robot to do x and our desire needs to be communicated
We can’t tell it that through a 1:1 interface
NOT reading of intention through cues
Communication still exists, and is expressly meant for the robot
Communicates intention directly with an interface which reads it from the user
ditto

4. Why Do It?
There are many users who cannot effectively communicate any other way
Paralysis
Amputation
Effective and non-intrusive reading of intention helps facilitate identity
Surrogacy
Sometimes a task must be very closely controlled but an explicit interface would be cumbersome
Emergencies, extra limbs, helping hands

5. How Is it Done?
The two main places to read intention from are the user’s brain and muscles.
Brain-Computer Interfaces (BCI) look directly at the user’s brain (most often restricted to the motor cortex) to learn what motions the user wishes to do
Electromyograms (EMG) sense the activation signals emitted from nerves and muscles to learn what motions a user is attempting to do
Both can be invasive or noninvasively implemented
[1]
[2]

6. What Are We Doing Here?
Putting together a taxonomy to help sort through the current state of the art
Replace
Replace-In-Place
Augment
Sometimes a system is built which is meant for one user but could easily have a totally different application (and sometimes do it better)
1 minute

7. Replace
A user has lost a significant capability (in most cases, the voluntary use of all muscles controlled via spinal reflex), and requires a service robot
Service robot must be able to effectively Replace a capability, doing so externally to the user
Most cases will involve a manipulator arm on a static base
Split between degrees of shared autonomy
Some mobile robots exist
Most don’t do manipulation tasks

8. Replace
High Autonomy
Shared Autonomy
No Autonomy
[3]
[4]
[5]
[7]
[6]

9. Replace-In-Place
R-I-P is specifically an integrative process in which the manipulator becomes part of the human user, replacing a hand or arm with a robotic prosthetic
Usage often constrained by amount of residual limb
Almost exclusively EMG due to anthropological imperatives
Split between methods which seek to control maximum degrees of freedom and which seek to have the most varied number of grasps 3

10. Replace-In-Place Maximize controllable DoFs Maximize grasp library [8]
[9]
[10]
[11]
[12]

11. Augment
Augment seeks to take a fully functional user and add further functionality
Ability
Robots that act alongside humans or as part of humans to increase their capabilities
Presence
Robots meant to go where a user cannot because the area is dangerous, or robots which are surrogates for the user in situations where the robot is acting in place of the user in a day-to-day aspect. 5

12. Augment
Ability
Presence
[6]
[8]
[13]
[14]

13. Cross-Usage Half of Augment cases appeared elsewhere
Systems which had a specific user in mind when built, but were tested on able-bodied users, showing other possibilities
One system appeared in all three
Weak Replace because requires motion of upper limb, strong R-I-P as long as significant residual limb remains, strong Augment. Mixed visual with EMG sensor to fluidly move an arm 7

14. Dominant Methodologies
BCI for Replace and EMG for R-I-P
Replace users often have no voluntary muscle control, so BCI works for the most users
There is an anthropological philosophical imperative to replace a person’s limb with a limb that works in the most similar way, even if that way decreases functionality and makes it less like the functioning of the original limb

15. Where Do We Go From Here? Replace Better grasping Integrate Mobility
Finer resolution
Improved Shared autonomy
Borrow from R-I-P to use synergies and predefined grasps
Integrate Mobility
Most solutions either manipulator or mobile 8

16. Where Do We Go From Here? Replace-In-Place Put it all Together
Some solutions have very good control of arm, some of hand, almost none can do both
Keep working on the electrodes
Percutaneous is good progress
IMES is making a strong case for permanent intramuscular
sEMG integrated into clothing
Measurement of EMG activity with textile electrodes embedded into clothing
[15]

17. Where Do We Go From Here? Augment Keep working on the other two
A lot of good work in helping those who need it will produce results for the Augment market
Test degrees of embodiment
Some systems seemed to cause the user to identify with the robot. This needs to be investigated further to make safe surrogates
User acceptance
Do people want more arms?

18. Thanks for The Course

19. How Is it Done? Non-Invasive BCI
[1],[16],[17],[18]

20. How Is it Done? Non-Invasive BCI
Electroencephalogram (EEG)
Surface of skull
Noisy
Reads electrical impulses
[19]

21. How Is it Done? Non-Invasive BCI
Functional Magnetic Resonance Imaging (fMRI)
Reads BOLD
Blood-oxygen-level dependence
Laggy
Highly precise
Immobile
Magnetoencephalography(MEG)
Reads magnetic fields around brain
Stronger signal than electrical signals

22. How Is it Done? Non-Invasive BCI
Near-Infrared Spectroscopy (NIRS)
Surface of skull
Reads BOLD
Array of infrared LEDs and receivers
Highly influenced by placement
[20],[21]

23. How Is it Done? Non-Invasive BCI
PET (positron emission tomography), SPECT (single positron emission computed tomography), CAT (computerized axial tomography)
Immobile
Uses radioactive fluid introduced into bloodstream to read blood flow
fNIRS (Functional NIRS
Often used interchangeably in literature with NIRS

24. How Is it Done? Invasive BCI
Intracortical Electrode
Very high resolution
Down to the individual neuron
Usually 96-channel
Electrocorticography (ECoG)
Similar to EEG, but on
surface of brain instead of skull
Very low noise
[1]
[22]
[23]

25. How Is it Done? Non-Invasive EMG
Surface EMG (sEMG)
Attached to skin of arm
Noise problems
Cross-talk between muscles limits maximum number of sites
Can use mixture of multiple muscles to get a synergistic input
[24]

26. How Is it Done? Invasive EMG
Targeted Muscle Reinnervation (TMR)
Surgery to add sites by reinnervating muscles using legacy nerves
Works with sEMG by adding usable sites
[25]

27. How Is it Done? Invasive EMG
Percutaneous
Pierces through the first layer of skin, but not does not pass through the subcutis
Inserted in the same way as a microdermal body piercing
Creates a dedicated, fixed site which has much lower attenuation than found in sEMG
[26]

28. How Is it Done? Invasive EMG
Intramuscular
Implanted into muscle
High accuracy
Wired and wireless
Can get more varied signals
No crosstalk
[27]

29. Outliers Sometimes a system didn’t quite fit
Pieces of R-I-P systems were in R-I-P, but could not work well on their own
Were disqualified from Augment because of that
Percutaneous R-I-P has no real implementation yet
Intramuscular can be very portable, but not very easy to get
BCIs with too much autonomy are of limited use outside of controlled environments

30. 1
Brain–Machine Interfaces: Basis and Advances, Becedas, Jonathan 2
Effect of clinical parameters on the control of myoelectric robotic prosthetic hands, Atzori et. al 3
Demonstration of a Semi-Autonomous Hybrid Brain–Machine Interface Using Human Intracranial EEG, Eye Tracking, and Computer Vision to Control a Robotic Upper Limb Prosthetic, Mcmullen et. al 4
Control of a humanoid robot by a noninvasive brain–computer interface in humans, et. al 5
Autonomy Infused Teleoperation with Application to BCI Manipulation, et al. 6
Proportional Myoelectric Control of Robots: Muscle Synergy Development Drives Performance Enhancement, Retainment, and Generalization, Ison et. al 7
Neurally controlled robotic arm enables tetraplegic patient to drink coffee of her own volition, Hochberg et. al 8
A Human-Assisting Manipulator Teleoperated by EMG Signals and Arm Motions, Fukuda, et. al 9
First-in-Man Demonstration of Fully Implanted Myoelectric Sensors for Control of an Advanced Electromechanical Arm by Transradial Amputees, Pasquina et. al 10
Real-time myoelectric control of a multi-fingered hand prosthesis using principal components analysis, Matrone et. al 11
Novel postural control algorithm for control of multifunctional myoelectric prosthetic hands, Segil et. al 12
Surface EMG in advanced hand prosthetics, Castellini et. al 13
fMRI-Based Robotic Embodiment: Controlling a Humanoid Robot by Thought Using Real-Time fMRI, Cohen et. al

31. 14
Comparative Study of SSVEP- and P300- Based Models for the Telepresence Control of Humanoid Robots, Zhao et. al 15
16,17,18
The potential of positron-emission tomography to study anticancer-drug resistance, West, Catharine M. L; Jones, Terry; Price, Pat; Single Photon Emission Computed Tomography Market – Global Industry Analysis, Size, Share, Growth, Trends and Forecast 2016 – 2020, MEDAGADGET; computerized axial tomography: computed X-ray tomography (CT) scanner, Britannica 19
Brain Computer Interfaces, a Review, Nicolas-Alonso, Luis Fernando; Gomez-Gil, Jaime
20-21
NirX; NIRS: A Head Patch That Monitors Brain Blood Flow and O2, Santos, Glenn 22
Utility of electrocorticography in the surgical treatment of cavernomas presenting with pharmacoresistant epilepsy, San-Juan et. al 23
Intracortical Visual Prosthesis, Laboratory of Neural Prosthetic Research at the Illinois Institute of Technology 24
A comparison of the real-time controllability of pattern recognition to conventional myoelectric control for discrete and simultaneous movement, Young, Aaron J; Smith, Lauren H; Rouse, Elliott J; Hargrove, Levi J 25
Targeted Muscle Reinnervation and Advanced Prosthetic Arms, Cheesborough, Jennifer E.; Smith, Lauren H.; Kuiken, Todd A.; Gregory A. Dumanian 26
A Novel Percutaneous Electrode Implant for Improving Robustness in Advanced Myoelectric Control, M.Hahne, Janne; DarioFarina; NingJiang; DavidLiebetanz 27
Use of Intramuscular Electromyography for the Simultaneous Control of Multiple Degrees of Freedom in Upper-Limb Myoelectric Prostheses, Smith, Lauren Hart