Estimating Physical Intent For Exoskeleton Control or Reading The Operator’s Mind Chris Atkeson, CMU

Estimating Physical Intent Anticipate movement We do it all the time: martial arts, partner dance, … “Tells” in poker: Gaze, head and body movement, facial expression, … Feel or see muscles tense, postural change, … Suppose we could instrument our partner/opponent? What signals would we go after? What sensors would we use? Predict muscle forces, and body movement. Predict a state at a time: x(t), a trajectory, or a probability distribution: p(x(t)) Estimate/track human decision process: what behavior is chosen, with what parameters? What utility function is the human optimizing? Uses: exoskeleton control system; online operator situation awareness; cognitive assistance and physical guidance; command and squad visualization; situation awareness, coordination, and cognitive assistance; after-action reports; and new operator training and operator learning from practice and coaching;

Time Scales 10 sec - Behavior selection. Rely on perceiving situation and thinking like your operator and your opponents. What are the probabilities of various attacks? Responses? 1 sec - Behavior selection and reflex responses. Measure set/tells to estimate probability of particular behaviors and reflexes. Maintain probability distribution of possible behaviors and reflexes. 100 msec - Muscle activation. EMG, muscle activation modeling force helpful here. 10 msec - Biomechanics. Joint position, velocity, contact forces, internal muscle and tissue forces. 1 msec - Local and global accelerations and local deformation dynamics. Distributed MEMs IMUs on operator sensing local accelerations, angular velocities, vibration, impacts, shock waves, and other high frequency mechanical phenomenon.

Signals perceive what the operator sees, hears, feels, and smells. outward facing superhuman perception (RF signals, UV, IR, ultrasound imaging at contacts, ...) brain signals, spinal signals (motorneuron pools), motor nerve signals, sensory nerve signals muscle electrical signals (EMG) muscle force signals (FMG), muscle force at tendons, muscle internal pressure implanted tissue markers, other implanted sensors tissue deformation tracked by tissue imaging (ultrasound, optical) operator internal forces operator-exoskeleton contact force operator-exoskeleton contact deformation, deformation rate train user to emit special signals (play a videogame with their body, hands, eyes, neural signals, muscle signals, ...)

Timing (changing load in hand) Fast reflexes (involuntary) - 30-50ms. “Long loop reflexes" (can be voluntarily enabled/disabled) - 50-80ms. Triggered reactions - 80-120ms. Reaction to an object slipping in a grasp is about 80ms. [Johansson and Westling 1984] Voluntary responses - 120-180ms. Visual reaction times in the 200ms range. Typically, (full) force production trails EMG by about 30ms in fast muscles (electromechanical delay, due to calcium ion dynamics traveling from sarkoplasmatic reticulum to cross bridge binding sites.

Implanted EMG electrodes (similar to what we put in pets) http://neural.iit.edu/research/imes/ http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4716669 http://link.springer.com/chapter/10.1007%2F978-3-319-08072-7_12 http://www.ncbi.nlm.nih.gov/pubmed/19224729 http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2878855/

Muscle force Implanted pressure sensor Ultrasound tracking of tendon stretch Optic fiber-based tendon strain measurement http://jap.physiology.org/content/99/2/603 Finni T, Komi PV, and Lukkariniemi J. Achilles tendon loading during walking: application of a novel optic fiber technique. Eur J Appl Physiol Occup Physiol 77: 289–291, 1998. Komi PV, Belli A, Huttunen V, Bonnefoy R, Geyssant A, and Lacour JR. Optic fibre as a transducer of tendomuscular forces. Eur J Appl Physiol Occup Physiol 72: 278–280, 1996.

Muscle movement Implanted sensors measuring distance Continuous imaging Ultrasound (transmission vs. reflective? implanted?) http://www.jove.com/video/50595/a-novel-application-of-musculoskeletal-ultrasound-imaging

Soft contact force sensors (Yong Lae Park, CMU)