Bilateral Teleoperation of Multiple Cooperative Robots over Delayed Communication Network: Theory Dongjun Lee Mark W. Spong Research partially supported by the Office of Naval Research (N and N ), the National Science Foundation (IIS and CCR ), and the College of Engineering at the University of Illinois.

Outline 1. Motivations 2. Problem Formulation 3. Passive Decomposition of Slave Robots 4. Control Design 5. Conclusions Part II: Simulation and Semi-Experiment

Motivations Applications: 1. Space Structure Construction/Maintenance - Hubble telescopes, International Space Station,… 2. Remote Construction/Maintenance of Civil Structures - Bridge, Highway, Tall buildings,… 3. Operations in Hazardous Environments - Nuclear plants, Deep water, … Bilateral Teleoperation - Human’s intelligent intervention in uncertain environments Multi-Robot Cooperation - Mechanical strength and dexterity - Robustness and safety Bilateral Teleoperation of Multiple Cooperative Robots

Challenges and Requirements 1. Abstraction - human is able to operate only small DOF simultaneously 2. Secure grasping - no dropping of the grasped object 3. Haptic feedback - crucial for manipulation tasks 4. Interaction safety and stability - stably coupled with humans, objects, and environments

Outline 1. Motivations 2. Problem Formulation 3. Passive Decomposition of Slave Robots 4. Control Design 5. Conclusions

Dynamics of Master and Multiple Slave Robots Dynamics of a single master (m-DOF) Dynamics of multiple slave robots (n 1 +n 2 +…+n N -DOF) n-DOF product system (n=n 1 +n 2 +…+n N -dimensional) Stack -up inertia Corioliscontrol human force velocity

Grasping Shape Function: Holonomic Constraints Grasping shape control objective desired (constant) grasping shape q1q1 q2q2 q3q3 m-dim. level sets - m-dim. holonomic constraints on the config. space of slave robots (m < n) - assumed to address the internal formation shape for cooperative grasping - smooth and full-rank Jacobian (i.e. smooth submersion) - overall group motion evolving on m-dim. level sets (submanifold) master’s DOF

Communication and Control (C&C) Structure - C&C delay between the master and the slaves - Centralized C&C module for multiple slaves - negligible delays among the slaves - workspaces of slaves are close to each other (e.g. cooperative grasping)

Semi-Autonomous Teleoperation Architecture Observation: - secure grasping is of foremost importance for safety - the system cannot be completely free from time-delay, i.e. system performance would be compromised in some aspects Semi-autonomous teleoperation: 1. local grasping control - secure grasping immune to communication-delay - autonomous control would be enough due to simplicity of cooperative grasping control objective 2. delayed bilateral teleoperation - communication-delay effect confined in bilateral teleoperation - sluggish response could be taken care of by intelligent humans - delayed teleoperation is relatively well-studied areas

Energetic Passivity for Safe/Stable Interaction - passive with total master/slave mechanical power as supply rate - stable interaction with any E-passive humans [Hogan] /objects/environments Energetic passivity total slave-ports mechanical power master-port mechanical power

Outline 1. Motivations 2. Problem Formulation 3. Passive Decomposition of Slave Robots 4. Control Design 5. Conclusions

Passive Decomposition of Multiple Slaves Robots The Passive Decomposition [Lee&Li, CDC03] decouples the locked and shape systems from each other while enforcing passivity - Can achieve tight/secure grasping regardless of overall group behavior - Ensure secure grasping and interaction stability simultaneously internal group coordination (cooperative grasping) Shape System behavior of overall group (and grasped object) Locked System Coupling: dropping object!!!

Orthogonal Decomposition w.r.t. Inertia Metric Locked system velocity v L : parallel w.r.t. the level sets of q E : (behavior of grasped object and total group) Shape system velocity v E : orthogonal complement w.r.t. inertia matrix (cooperative grasping) locked system velocity v L shape system velocity v E Grasping shape function Tangent space decomposition basis of kernel of q E basis of orthogonal space

Passive Decomposition of Slave Group Dynamics - Shape system ((n-m)-DOF) explicitly represents cooperative grasping shape q E (q) - Locked (m-DOF) system describes overall group behavior - Locked and shape dynamics are similar to usual mechanical systems: - M L (q), M E (q) : symmetric and positive-definite - M L (q)-2C L (q,q), M E (q)-2C E (q,q) : skew-symmetric - Coupling is energetically conservative: Passive Decoupling - C LE (q,q) =-C EL T (q,q) -> v L T C LE (q,q)q E + q E T C EL T (q,q)v L =0 - Power and kinetic energy are also decomposed Original Slave Dynamics Passive Decomposition Decomposed Dynamics

Energetic Structure of Decomposed Dynamics - We can decouple the shape system (cooperative grasping) and the locked system (overall group) from each other while enforcing passivity - Desired cooperative grasping and overall group behavior can be achieved simultaneously while enforcing interaction stability Original SystemDecomposed System passive decoupling

Outline 1. Motivations 2. Problem Formulation 3. Passive Decomposition of Slave Robots 4. Control Design 5. Conclusions

Semi-Autonomous Control Decomposed Dynamics Scattering-based teleoperation control for decoupled locked system Local grasping control control for decoupled shape system Passive decoupling Total Slave Control - Adjusting q E d, and PD-gains, fixtureless grasping can be achieved for flexible object - Although dynamics is decoupled, other effects (e.g. inertia of object) can still perturb the shape system via the internal force F E : feedforward cancellation is necessary Grasping Dynamics (Decoupled Shape System) internal force PD/FF-based Control estimate of internal force desired grasping shape Local Grasping Control

Scattering-Based Teleoperation of Locked System control human/combined external forces Dynamics of Master Robot and Slave Locked System (both are m-DOF) Shape system (locally controlled) Locked System (decoupled) By operating the master robot of manageably small DOF, human operators can tele-control the behavior of the grasped object over the delayed master-slave communication channel while perceiving combined external forces acting on the grasped object and slaves

Symmetric Scattering-Based Teleoperation: - scattering communication (to passify comm. delays) and impedance (PI) controls - asymptotic position-error convergence proof with Z=Kv (i.e. matching condition [Stramigioli et al, TRA03] ) : so far, only boundedness of position-error has been established. - force reflection in static manipulation (negligible acceleration/velocity) Impedance Control (PI-Control) line impedance (user-specific) Scattering Variables (Power Decomposition) reflected (from comm.) incident (to comm.) Scattering-Based Symmetric Teleoperation

Conclusions We propose a control framework for bilateral teleoperation of multiple cooperative robots over delayed master-slave comm. channel: - passive decomposition: the decoupled shape (cooperative grasping) and locked (behavior of the grasped object) systems - local grasping control for the shape system: high precision cooperative grasping regardless of human command/comm. delays - scattering-based bilateral teleoperation of the locked system: human can tele-control behavior of the cooperatively grasped object by operating a small-DOF of the master robot, while perceiving combined force on the slaves and the grasped object over the delayed communication channel - enforce energetic passivity: interaction safety and stability are enhanced Part II will present simulation and semi-experiment results.