Bilateral Teleoperation of Multiple Cooperative Robots over Delayed Communication Network: Application Dongjun Lee Mark W. Spong Oscar Martinez-Palafox Research partially supported by the Office of Naval Research (N and N ), the National Science Foundation (IIS and CCR ), and the College of Engineering at the University of Illinois.

Outline 1. Review of the Proposed Control Framework 2. Simulation Results 3. Semi-Experimental Results 4. Conclusions

Bilateral Teleoperation of Cooperative Multi-Robots Combine advantages of - bilateral teleoperation: human intervention in uncertain environments - multi-robot cooperation: mechanical strength/dexterity & robustness/safety - applications: remote construction/maintenance of space/under-water/civil structures in possibly hazardous environments

Semi-Autonomous Teleoperation - Passive Decomposition [Lee&Li, CDC03] decomposes slave dynamics into decoupled shape (formation shape) and locked (overall group motion) systems - Local grasping control of decoupled shape system: secure/tight grasping regardless of human command via delayed comm. Channel - Bilateral teleoperation of locked system: by operating the master robot of manageably small DOF, human can tele-control the behavior of the grasped object over the delayed comm. channel while perceiving external forces internal formation shape (cooperative grasping) Shape System behavior of overall group (and grasped object) Locked System Coupling: dropping object!!! Passive decoupling

System Modelling and Grasping Shape Function Dynamics of a single master (m-DOF) Dynamics of multiple slave robots (n 1 +n 2 +…+n N -DOF) n-DOF product system (n=n 1 +n 2 +…+n N -dimensional) Stack -up inertia Corioliscontrolhuman force velocity grasping shape function describes internal group formation shape Grasping Shape Function: R n →R n-m desired (constant) grasping shape master’s DOF

locked system shape system Passive Decomposition and Local Grasping Control Decomposed Slave Dynamics passive decoupling Local Grasping Control FF cancellation of internal force: although dynamics is decoupled, other effects (e.g. object’s inertia) can still perturb the shape system through internal force F E desired grasping shape Locked system: abstracts overall behavior of multiple slave robots and grasped object Shape system: describes internal group formation of slave robots (i.e. cooperative grasping)

Scattering-Based Teleoperation of Locked System control human/combined external forces Dynamics of Master Robot and Slave Locked System (both are m-DOF) Shape system (locally controlled) Locked System Scattering-Based Teleoperation of Locked system: - humans can tele-control the behavior of the grasped object over delayed comm. channel while perceiving external forces acting on the object and slaves - asymptotic position coordination/static force reflection

Outline 1. Review of the Proposed Control Framework 2. Simulation Results 3. Semi-Experimental Results 4. Conclusions

Simulation Settings - grasping shape function is defined s.t. three slaves form an equilateral triangle (w/ side length L) whose rotation is specified by the heading of agent 2 - human operator can tele-control the position and rotation of the triangle by operating 3-DOF master robot (translation and yaw) - 10% identification errors for inertias of robots (nominal: m=1kg, I=1kgm 2 ) Delay 0.5s 3-DOF Master (x,y)-translation yaw rotation Three 3-DOF Slave Robots deformable object (no friction) agent1 agent2 agent3

Simulation: Importance of Decoupling - no grasped object (just motion coordination) w/ PD-based grasping control - without decoupling control, grasping shape (i.e. shape system) is perturbed by human command and overall group behavior - slight grasping shape distortion w/ decoupling is due to inertial uncertainty Without Passive Decoupling Control With Passive Decoupling Control

Simulation: Heavy Object Fixtureless Manipulation - even if dynamics is decoupled, inertial effect of object (w/ frictionless contact) perturbs cooperative grasping through the internal force F E - this perturbation can be cancelled out by feedforward cancellation of the internal force F E (or also by large enough PD-gains) With Feedforward Cancellation of Internal Force Without Feedforward Cancellation of Internal Force

Heavy Object Manipulation: Contact/Human Force - human can perceive the total inertias of the grasped object and the slave robots - human can also perceive sensation of grasping loss - better load-balancing is achieved w/ FF-cancellation of the internal force F E, as grasping shape becomes more rigid good load balance due to grasping rigidity due to grasping shape deformation

Simulation: Force Reflection - external forcing (x-direction) on the grasped object is faithfully reflected to the human operator (i.e. haptic feedback) - load balancing among slaves is degraded as the grasped object is deformed in the rigidly-maintained grasping shape Three 3-DOF Slave Robots deformable object agent1 agent2 agent3 external force due to object’s deformation human force

Outline 1. Review of the Proposed Control Framework 2. Simulation Results 3. Semi-Experimental Results 4. Conclusions

Semi-Experiment Setting - three slave robots: 2-DOF point mass dynamics (only x,y translations) - Phantom Desktop is used as master with its workspace constrained on (x,y)-plane - Grasping shape function: : specifies rotation and shape of the triangle formed by the three slaves Delay 0.5s 2-DOF Master PHANToM Desktop: constrained on plane (i.e. (x,y)-translation) Three 2-DOF Slave Robots agent1 agent2 agent3 deformable object external force

Semi-Experiment: Deformable Object Manipulation - x-directional motion (full-range) w/ fixtureless grasping - grasping security is preserved regardless of human command - human can perceive the combined inertia of slaves and grasped object - increase of some slaves' contact force due to inertia/deformation of object secure/precise grasping w/ FF-term due to object deformation human perceives inertias of object/slaves

Semi-Experiment: Obstacle Perception - external force (x-direction) on the grasped object center - force generated by the PI-action in the local impedance controls - object’s deformation again leads in unbalanced load sharing among slaves human perceives external force Secure/precise grasping w/ FF-term due to object deformation

Conclusions We propose a control framework for bilateral teleoperation of multiple cooperative robots over delayed master-slave comm. channel: - passive decomposition: the decoupled shape (cooperative grasping) and locked (behavior of the grasped object) systems - local grasping control for the shape system: high precision cooperative grasping regardless of human command/comm. delays - scattering-based bilateral teleoperation of the locked system: human can tele-control behavior of the cooperatively grasped object by operating a small-DOF of the master robot, while perceiving combined force on the slaves and the grasped object over the delayed comm. channel - enforce energetic passivity: interaction safety and stability - Semi-experiment and simulation results are presented and validate efficacy of the proposed control framework Possible impacts on emerging or traditional applications: - remote construction/maintenance of space/under-water/civil structures in hostile/hazardous environments