Design of a Multi-Threaded Distributed Telerobotic Framework Mayez Al-Mouhamed, Onur Toker, and Asif Iqbal Mayez Al-Mouhamed, Onur Toker, and Asif Iqbal College of Computer Science and Engineering King Fahd University of petroleum and Minerals Dhahran Kingdom of Saudi Arabia

OUTLINE Background on Telerobotics Background on Telerobotics A multi-threaded distributed framework A multi-threaded distributed framework Telerobotic server components Telerobotic server components – PUMA component – Force component – Component hierarchy – Integrated scheme Telerobotic client components Telerobotic client components – Integrated scheme Multi-threaded distributed telerobotic system Multi-threaded distributed telerobotic system Conclusion Conclusion

Background on telerobotics Telerobotics is an approach that extends a human operator’s sensing and manipulating capabilities to a remote environment. Telerobotics has numerous applications in hazardous or hostile environments. The purpose of this work is to design a reliable and efficient man- machine interface between a slave arm (robot) and a master arm over a network which preserves a large degree of operator dexterity. Computer vision and force feed techniques are implemented to enhance the maneuverability of the operator by providing him force feeling. To avoid communication delays in frequently interacting telerobotic systems, the operator will have a library of local automation tasks. Only coarse operator supervision is required. For example, picking up a tool, moving tool to a certain location, returning tool to initial position, etc. Telerobotics is a multidisciplinary area integrating knowledge from robotics, real-time operating systems and programming, network programming, 3D stereo vision, mechanical linkage engineering, ergonomics, etc.

A Multi-Threaded Distributed Framework Client-Server distributed component telerobotic system. Client-Server distributed component telerobotic system. A telrobotic server has components (PUMA, Force Sensor, and Decision Server) and interfaces (Proxy Robot, Sensor, and DecisionServer). A telrobotic server has components (PUMA, Force Sensor, and Decision Server) and interfaces (Proxy Robot, Sensor, and DecisionServer). One or more telerobotic client components One or more telerobotic client components An integrated scheme of client-server components An integrated scheme of client-server components A multi-threaded distributed telerobotic system A multi-threaded distributed telerobotic system

Telerobotic server components PUMA Component PUMA Component Acts as a software proxy (thread) of the slave robot arm diagram. Acts as a software proxy (thread) of the slave robot arm diagram. Public methods Public methods ConnectRobot, ConnectRobot, InitializeRobot, InitializeRobot, incremental joint and Cartesian motion commands, incremental joint and Cartesian motion commands, tool and world frame motion coordination, etc. tool and world frame motion coordination, etc. Public properties Public properties Booleans that represent robot state. Booleans that represent robot state. Public Events Public Events motion commands motion commands communication and synchronization mechanisms. communication and synchronization mechanisms.

PUMA component

Force Sensor Component Force Sensor Component A service thread for a wrist force sensor attached to slave robot arm (3 force+3 moments) A service thread for a wrist force sensor attached to slave robot arm (3 force+3 moments) Provides reflected force feedback from slave arm (server) to operator (client) Provides reflected force feedback from slave arm (server) to operator (client) Public properties Public properties start and stop reading start and stop reading TimerValue, etc. TimerValue, etc. SensorThreadPriority, SensorThreadPriority, Real-time streaming of force data from server to client Real-time streaming of force data from server to client Telerobotic server components

Component hierarchy

Integrated scheme - server

Telerobotic Client Components Client GUI (Graphic User Interface) for remote testing and maintenance operations Client GUI (Graphic User Interface) for remote testing and maintenance operations IDecisionServerto interface to server through.NET Remoting IDecisionServerto interface to server through.NET Remoting All the definitions to execute methods on PUMA and ForceSensor components All the definitions to execute methods on PUMA and ForceSensor components After initialization, the client carries an empty un- referenced copy of IDecisionServer After initialization, the client carries an empty un- referenced copy of IDecisionServer Following the network connection, the client can reference any instance of DecisionServer Following the network connection, the client can reference any instance of DecisionServer

Integrated scheme - client

Integrated Scheme of Client-Server Components Inherits DecisionServer from IDecisionServer, IProxyRobot, IForceSensor interfaces Inherits DecisionServer from IDecisionServer, IProxyRobot, IForceSensor interfaces To invoke DecisionServer events, Shim Classes are used as agents to forward DecisionServer events to client. To invoke DecisionServer events, Shim Classes are used as agents to forward DecisionServer events to client.

A Multi-threaded distributed Telerobotic System Simultaneous activation of many threads like Simultaneous activation of many threads like Two digital cameras generate stereo pictures which are sent to the client Two digital cameras generate stereo pictures which are sent to the client Grabbing and transfer of stereo video data Grabbing and transfer of stereo video data Reading and transferring force sensor data Reading and transferring force sensor data Sending and receiving robot control signals over the LAN to one or more clients Sending and receiving robot control signals over the LAN to one or more clients Both the stereo data and the distributed component calls share the same LAN using different ports for data transfer Both the stereo data and the distributed component calls share the same LAN using different ports for data transfer

Integrated system DecisionServer component provides slave supervisory control. DecisionServer component provides slave supervisory control. Server Side Interfaces and.NET Remoting Server Side Interfaces and.NET Remoting A set of definitions of public methods and properties. A set of definitions of public methods and properties. Servers as a contract for any component that implements this interface. Servers as a contract for any component that implements this interface. Allows hiding the actual component to increase security. Allows hiding the actual component to increase security. Uses IProxyRobot and IForceSensor to communicate with PUMA and Force Sensor components Uses IProxyRobot and IForceSensor to communicate with PUMA and Force Sensor components IDecisionServer inherits both of the above interfaces. IDecisionServer inherits both of the above interfaces. Allows defining a unified set of methods, properties, and events within DecisionServer component. Allows defining a unified set of methods, properties, and events within DecisionServer component..NET Remoting publishes an instance of DecisionServer component on the LAN..NET Remoting publishes an instance of DecisionServer component on the LAN..NET Remoting enables access to remote objects using SOAP..NET Remoting enables access to remote objects using SOAP.

Events forwarding using Shim classes

Server system of distributed framework

Client system of distributed framework

Conclusion A reliable and efficient man-machine interface between a slave robot (server) and a master arm (client) over a network while preserving some degree of operator dexterity A reliable distributed components (.NET Remoting) is implemented: Software reusability, ease of extensibility, debugging, and data Encapsulation Advanced software tools like.NET Framework (comp. to DCOM) The components communicate using.NET Remoting and SOAP Automatic handling of network resources and data transfer Isolation of components from network protocol issues Enhances data security as well as facilitates deployment Multi-threaded execution for multi-streaming of force, command and stereo data.