Non-Explicit Communication with Robotic Manipulators: A Review Paul M. Calhoun

What Is It? Communication with robots that is both explicit and direct is well defined Joystics, words, gestures, are all both explicit and direct Communication which is non-explicit, and non-direct is also well defined The robot sees that an event has occurred or an object is in a location and takes action based on that Keep it VERY short

What Is It? Non-Explicit but direct communication is rare We want the robot to do x and our desire needs to be communicated We can’t tell it that through a 1:1 interface NOT reading of intention through cues Communication still exists, and is expressly meant for the robot Communicates intention directly with an interface which reads it from the user ditto

Why Do It? There are many users who cannot effectively communicate any other way Paralysis Amputation Effective and non-intrusive reading of intention helps facilitate identity Surrogacy Sometimes a task must be very closely controlled but an explicit interface would be cumbersome Emergencies, extra limbs, helping hands

How Is it Done? The two main places to read intention from are the user’s brain and muscles. Brain-Computer Interfaces (BCI) look directly at the user’s brain (most often restricted to the motor cortex) to learn what motions the user wishes to do Electromyograms (EMG) sense the activation signals emitted from nerves and muscles to learn what motions a user is attempting to do Both can be invasive or noninvasively implemented [1] [2]

What Are We Doing Here? Putting together a taxonomy to help sort through the current state of the art Replace Replace-In-Place Augment Sometimes a system is built which is meant for one user but could easily have a totally different application (and sometimes do it better) 1 minute

Replace A user has lost a significant capability (in most cases, the voluntary use of all muscles controlled via spinal reflex), and requires a service robot Service robot must be able to effectively Replace a capability, doing so externally to the user Most cases will involve a manipulator arm on a static base Split between degrees of shared autonomy Some mobile robots exist Most don’t do manipulation tasks

Replace High Autonomy Shared Autonomy No Autonomy [3] [4] [5] [7] [6]

Replace-In-Place R-I-P is specifically an integrative process in which the manipulator becomes part of the human user, replacing a hand or arm with a robotic prosthetic Usage often constrained by amount of residual limb Almost exclusively EMG due to anthropological imperatives Split between methods which seek to control maximum degrees of freedom and which seek to have the most varied number of grasps 3

Replace-In-Place Maximize controllable DoFs Maximize grasp library [8] [9] [10] [11] [12]

Augment Augment seeks to take a fully functional user and add further functionality Ability Robots that act alongside humans or as part of humans to increase their capabilities Presence Robots meant to go where a user cannot because the area is dangerous, or robots which are surrogates for the user in situations where the robot is acting in place of the user in a day-to-day aspect. 5

Augment Ability Presence [6] [8] [13] [14]

Cross-Usage Half of Augment cases appeared elsewhere Systems which had a specific user in mind when built, but were tested on able-bodied users, showing other possibilities One system appeared in all three Weak Replace because requires motion of upper limb, strong R-I-P as long as significant residual limb remains, strong Augment. Mixed visual with EMG sensor to fluidly move an arm 7

Dominant Methodologies BCI for Replace and EMG for R-I-P Replace users often have no voluntary muscle control, so BCI works for the most users There is an anthropological philosophical imperative to replace a person’s limb with a limb that works in the most similar way, even if that way decreases functionality and makes it less like the functioning of the original limb

Where Do We Go From Here? Replace Better grasping Integrate Mobility Finer resolution Improved Shared autonomy Borrow from R-I-P to use synergies and predefined grasps Integrate Mobility Most solutions either manipulator or mobile 8

Where Do We Go From Here? Replace-In-Place Put it all Together Some solutions have very good control of arm, some of hand, almost none can do both Keep working on the electrodes Percutaneous is good progress IMES is making a strong case for permanent intramuscular sEMG integrated into clothing Measurement of EMG activity with textile electrodes embedded into clothing [15]

Where Do We Go From Here? Augment Keep working on the other two A lot of good work in helping those who need it will produce results for the Augment market Test degrees of embodiment Some systems seemed to cause the user to identify with the robot. This needs to be investigated further to make safe surrogates User acceptance Do people want more arms?

Thanks for The Course

How Is it Done? Non-Invasive BCI [1],[16],[17],[18]

How Is it Done? Non-Invasive BCI Electroencephalogram (EEG) Surface of skull Noisy Reads electrical impulses [19]

How Is it Done? Non-Invasive BCI Functional Magnetic Resonance Imaging (fMRI) Reads BOLD Blood-oxygen-level dependence Laggy Highly precise Immobile Magnetoencephalography(MEG) Reads magnetic fields around brain Stronger signal than electrical signals

How Is it Done? Non-Invasive BCI Near-Infrared Spectroscopy (NIRS) Surface of skull Reads BOLD Array of infrared LEDs and receivers Highly influenced by placement [20],[21]

How Is it Done? Non-Invasive BCI PET (positron emission tomography), SPECT (single positron emission computed tomography), CAT (computerized axial tomography) Immobile Uses radioactive fluid introduced into bloodstream to read blood flow fNIRS (Functional NIRS Often used interchangeably in literature with NIRS

How Is it Done? Invasive BCI Intracortical Electrode Very high resolution Down to the individual neuron Usually 96-channel Electrocorticography (ECoG) Similar to EEG, but on surface of brain instead of skull Very low noise [1] [22] [23]

How Is it Done? Non-Invasive EMG Surface EMG (sEMG) Attached to skin of arm Noise problems Cross-talk between muscles limits maximum number of sites Can use mixture of multiple muscles to get a synergistic input [24]

How Is it Done? Invasive EMG Targeted Muscle Reinnervation (TMR) Surgery to add sites by reinnervating muscles using legacy nerves Works with sEMG by adding usable sites [25]

How Is it Done? Invasive EMG Percutaneous Pierces through the first layer of skin, but not does not pass through the subcutis Inserted in the same way as a microdermal body piercing Creates a dedicated, fixed site which has much lower attenuation than found in sEMG [26]

How Is it Done? Invasive EMG Intramuscular Implanted into muscle High accuracy Wired and wireless Can get more varied signals No crosstalk [27]

Outliers Sometimes a system didn’t quite fit Pieces of R-I-P systems were in R-I-P, but could not work well on their own Were disqualified from Augment because of that Percutaneous R-I-P has no real implementation yet Intramuscular can be very portable, but not very easy to get BCIs with too much autonomy are of limited use outside of controlled environments

1 Brain–Machine Interfaces: Basis and Advances, Becedas, Jonathan 2 Effect of clinical parameters on the control of myoelectric robotic prosthetic hands, Atzori et. al 3 Demonstration of a Semi-Autonomous Hybrid Brain–Machine Interface Using Human Intracranial EEG, Eye Tracking, and Computer Vision to Control a Robotic Upper Limb Prosthetic, Mcmullen et. al 4 Control of a humanoid robot by a noninvasive brain–computer interface in humans, et. al 5 Autonomy Infused Teleoperation with Application to BCI Manipulation, et al. 6 Proportional Myoelectric Control of Robots: Muscle Synergy Development Drives Performance Enhancement, Retainment, and Generalization, Ison et. al 7 Neurally controlled robotic arm enables tetraplegic patient to drink coffee of her own volition, Hochberg et. al 8 A Human-Assisting Manipulator Teleoperated by EMG Signals and Arm Motions, Fukuda, et. al 9 First-in-Man Demonstration of Fully Implanted Myoelectric Sensors for Control of an Advanced Electromechanical Arm by Transradial Amputees, Pasquina et. al 10 Real-time myoelectric control of a multi-fingered hand prosthesis using principal components analysis, Matrone et. al 11 Novel postural control algorithm for control of multifunctional myoelectric prosthetic hands, Segil et. al 12 Surface EMG in advanced hand prosthetics, Castellini et. al 13 fMRI-Based Robotic Embodiment: Controlling a Humanoid Robot by Thought Using Real-Time fMRI, Cohen et. al

14 Comparative Study of SSVEP- and P300- Based Models for the Telepresence Control of Humanoid Robots, Zhao et. al 15 http://www.advancertechnologies.com/2013/03/diy-conductive-fabric-electrodes.html 16,17,18 The potential of positron-emission tomography to study anticancer-drug resistance, West, Catharine M. L; Jones, Terry; Price, Pat; Single Photon Emission Computed Tomography Market – Global Industry Analysis, Size, Share, Growth, Trends and Forecast 2016 – 2020, MEDAGADGET; computerized axial tomography: computed X-ray tomography (CT) scanner, Britannica 19 Brain Computer Interfaces, a Review, Nicolas-Alonso, Luis Fernando; Gomez-Gil, Jaime 20-21 NirX; NIRS: A Head Patch That Monitors Brain Blood Flow and O2, Santos, Glenn 22 Utility of electrocorticography in the surgical treatment of cavernomas presenting with pharmacoresistant epilepsy, San-Juan et. al 23 Intracortical Visual Prosthesis, Laboratory of Neural Prosthetic Research at the Illinois Institute of Technology 24 A comparison of the real-time controllability of pattern recognition to conventional myoelectric control for discrete and simultaneous movement, Young, Aaron J; Smith, Lauren H; Rouse, Elliott J; Hargrove, Levi J 25 Targeted Muscle Reinnervation and Advanced Prosthetic Arms, Cheesborough, Jennifer E.; Smith, Lauren H.; Kuiken, Todd A.; Gregory A. Dumanian 26 A Novel Percutaneous Electrode Implant for Improving Robustness in Advanced Myoelectric Control, M.Hahne, Janne; DarioFarina; NingJiang; DavidLiebetanz 27 Use of Intramuscular Electromyography for the Simultaneous Control of Multiple Degrees of Freedom in Upper-Limb Myoelectric Prostheses, Smith, Lauren Hart