2. Contents  Teleoperated robotic systems  The effect of the communication delay on teleoperation  Data transfer rate control for teleoperation systems  Experimental measurements  Conclusions 2

3. Teleoperated robotic systems  Telerobotics is the area of robotics concerned with the control of robots from a distance, chiefly using wireless connections (like Wi-Fi, Bluetooth, and similar), or the Internet.  A remote manipulator, is a device which, through a communication medium, allows a mechanism to be controlled by a human operator. The purpose of such a device is usually to move or manipulate hazardous materials for reasons of safety.  Distant controlled mobile robots is also an important application area of the teleoperation. 3

4. 4  It consists of a Master device and a Slave robot  Unilateral teleoperation: the operator moves the slave (generally low power robot), the master follows the motion of the slave  Bilateral teleoperation: the master (when it is in contact with the environment) reflects back the contact force to the slave, which is reflected to the operator Early teleoperation system (no network) Teleoperated robotic systems

5. 5

6. Telepresence  When sufficient amount of sensor information (vision, sound, force) is brought from the teleoperator site to the operator he or she feels physically present in the teleoperator site  Called also tele-existence  Important information is transferred and dangerous/noise is filtered 6

7. Teleoperation ower networks 7 Operator Haptic Device Communication Network Master robot Camera and Environment Display

8. Teleoperation ower networks The main goals during teleoperation system design:  Maintain the stability of the teleoperation irrespective of the behavior of the operator or the environment.  Provide a good transparency for the system - position tracking and force reflection 8

9. Instability due to Communication Delay FIRST9

10. Instability due to Communication Delay 10

11. How to deal with the instability problem?  Extend the control software of the master and slave robots with such stabilizer algorithms, that can assure the stability in the presence of delay.  However, these stabilizers always modify the received signals, compromising the transparency.  Try to reduce the communication delay and jitter. 11

12. Teleoperation systems over WLAN  PCh – Position channel Master  Slave  FCh – Force channel Slave  Master  VCh – Video channel Slave  Master  DCh – Other data channels in the same WLAN 12 Master Slave

13. The concept of feedback control  e – controlled variable (process output)  u – control signal, actuates the controlled process  d – not measurable disturbance signal  r – reference signal, encodes the desired behavior of e 13

14. The concept of feedback control 14

15. Control design Our original problem is to keep the delay and its fluctuation in the communication medium as small as possible. We want to formulate it in a feedback control approach. Accordingly, we need:  Measurements at Controlled Process output (e)  Controlled actuation at Controlled Process input (u) 15

16. Measurements 16

17. Measurements 17

18. The control error  Define the following combined errors Here, E denotes the expected value.  The value above has to be kept as small as possible. 18

19. Actuation  In a teleoperation system the video stream from the slave to master is the data flow with the highest transfer rate.  By regulating the data transfer in VCh ( u ), the performances in the FCh and PCh can be adjusted.  The amount of video data can be modified using several approaches. the sending period over the video channels can be modified. the size of the sent video frames can be changed. the quality of the sent image can be modified. (image compression techniques) 19

20. Disturbances  The communication channels of other applications, which are independent of the teleoperation application, also compete with the channels of the teleoperation. They also have the same chance of transmission.  The cumulated transfer rate of the other channels is denoted by d. 20

21. Disturbances  In wireless networks the maximum available transfer rate of an end node depends on the measured radio signal strength between the end node and the access point. The maximum available transfer rate of an end node automatically changes over time to maintain a reliable link between the devices in the WLAN. This is done by the dynamic rate scaling algorithm which runs on the wireless access point and it is application independent.  If the slave moves further away from the access point, its maximum available transfer rate gets reduced by the scaling algorithm. The rate decrease is done incrementally to pre-defined levels in function of the signal strength. 21

22. Controller Design  During controller design it has to be taken into consideration that there are multiple objectives that have to be satisfied.  Firstly, the delay and the jitter in FCh and PCh should be kept under prescribed limits.  Secondly, as good video transmission quality as possible has to be provided for all the cameras in the system.  Accordingly, the controller design can be formulated as a multi-objective optimization problem. 22

23. Theoretical approach  Lexicographic optimization: consider a set of N objective functions: with (possible) constraint  Arrange the objective functions in order of importance. Formally, the problem can be defined as: 23

24. The objective functions 24

25. Assumptions 25

26.  Apply a gradient-like algorithm:  If the assumptions A1 and A2 hold, it can be shown that: 26

27. 27

28. The video rate control law 28

29. The developed software 29

30. Experimental measurements  To modify the video rate in VCh, JPEG compression was used.  Influence of the video transfer rate on the delay and jitter: 30

31. Experimental measurements 31

32. Experimental measurements  Moving robot – Video rate control not active 32

33. Experimental measurements  Moving robot – Video rate control active 33

34. Experimental measurements  Moving robot – Comparison 34

35. Extension for multiple cameras  Teleoperation with more than one camera 35  PCh - Position channel Master  Slave  FCh - Force channel Slave  Master  VChi – Video channel Slaves  Master  CChi – Channels for control signals Master  Video slaves

36. Extension for multiple cameras  The displayed video information for the human operator received from the different cameras have different importance level.  A possible priority setting for the different video channels is: p 0 - priority of the video channel from the slave robot (VCh0). p 1 < p 2 - priority of those VCh’s, in the visual range of which the tracked mobile robot is. p 2 < p 3 - priority of VCh’s from the other video slaves. 36

37. Experimental measurements  The control signals (video transfer rates) for different cameras: 37

38. Experimental measurements  Three cameras, video controllers not active. 38

39. Experimental measurements  Three cameras, video controllers active. 39

40. Conclusions  To assure both the stability and transparency stability in teleoperation systems reliable communication in the channels of the teleoperation system has to be assured.  We proposed a video rate control algorithm which modifies the transfer rate in the video channels of the teleoperation system based on measurements performed in the position and force channels.  The control algorithm assures the best video quality corresponding to the prescribed delay and jitter values and actual disturbance level.  All of the performed experiments show that the proposed stream controller performs well in different traffic conditions, assuring small delay and jitter in the position and force channels of the teleoperation system. 40