Teleoperation Interfaces

Introduction Interface between the operator and teleoperator! Teleoperation interface is like any other HMI H(mobile)RI = TI Also a HCI

Introduction mechanical manipulation – ”included” interface Modern methods Closed loop teleoperation – MMI Supervisory control – HCI Mixed interfaces

Interface types, Fong Direct Closed loop Multimodal/multisensor Multisensor/actuator control Supervisory control Intelligent telerobot Novel New ones

Interface - Direct Closed loop control Realtime operator decision making is needed Operator controls with hand controllers (like onboard) High bandwith low delay communication

Multimodal/multisensor Complex robot in dynamic situation Individual actuator control, graphical feedback, coordinated motion Integrated display with combined sensor information

Supervisory control Remarkable part of the control in the teleoperator end Teleoperator is capable for more or less autonomous task execution “task based teleoperation”

Control methods DirectSupervisory

Novel interfaces ”novel” is relative gestures gazes brainwaves muscle movements WEB interfaces multimodal supervisory

Telepresence When sufficient amount of sensor information (vision, sound, force) is brought from the teleoperator site to the operator he or she feels physically present in the teleoperator site Called also tele-existence Important information is transferred and dangerous/noise is filtered

Virtual presence (reality) Like tele-presence except the sensor infromation is generated artificially by computer(s) Simulators Games Models

Augmented reality Real information (usually image data) is mixed with additional virtual information Numerical information, real- time models, etc.

Telepresence Already camera monitor combination creates some level of presence more sophisticated system is called for in order to call it telepresence To provide a perfect telepresence, all human senses should be transmitted from the teleoperator site to the operator site vision, hearing and sense are relatively easy smell and taste are more complicated

Vision Humans get 90% of their perception “To see is to believe” eyes are very complex opto-mechanical systems FoV is (H)180 deg x (V)120 deg Focused area only few degrees Movements from whole area Extremely difficult system to be imitated

Vision Head tracking Head following cameras (2-3 DoF) HMD => relatively good feeling of presence

Hearing Human area 16 – 20000Hz Important in telepresence In case of mobile machine control the noise can be filtered and the important sounds transferred with reasonable volume

Touch The most important human sense Human touch sensors – mechanoreceptors – are activated by touch, i.e. by pressure on the tissues Two basic classes tactile information (“touch”) kinesthetic information (“force”)

Tactile referring to the sense of contact with the object, mediated by the responses of low-threshold mechanoreceptors innervating the skin (say, the finger pad) within and around the contact region

Kinesthetic referring to the sense of position and motion of limbs along with the associated forces conveyed by the sensory receptors in the skin around the joints, joint capsules, tendons, and muscles, together with neural signals derived from motor commands

Force feedback (kinesthetic) force generated by the teleoperator, usually a manipulator, is fed back to the operator in order to generate a real response in gripping and manipulation tasks Also in virtual environments Inbuilt in mechanical manipulators

Haptic feedback (tactile) haptic feedback, the tactile skin sensors have the main role. tactile sensing of the robot manipulator is fed back to the fingers of the operator Other possibilities also

Vestibular sensors inside the inner ear angular acceleration and thus rotation linear acceleration in the horizontal and vertical plane, i.e. to gravity => position and movements of the head to be detected Important in dynamic driving tasks

Vestibular feedback not usually used in teleoperation not needed and expensive to implement usually in simulators to create presence If vision and vestibular sensors mismatch => simulator sickness (=seasickness)

Simulator Sickness similar to motion sickness difference is that SS can occur without any actual motion of the operator Symptoms: apathy, general discomfort, headache, stomach awareness, nausea, etc. encountered especially when HMD type displays are used

Simulator Sickness The most typical reason of SS is the cue conflict In cue conflict different nerves get different information from the environment Typically conflict between visual and vestibular inputs especially when HMD is used and the time lags in vision and control Moving teloperator + =>

What is Presence Overview

Traditional explanation Based on tele(virtual)-presence Sheridan’s definition of telepresence

Traditional explanation Extent of sensory information has a much greater impact than the other two combined. These three factors however cannot describe presence alone. Task variables, such as task difficulty and degree of automation, also are important to presence.

Is presence only sensor information? It’s very easy to be unpresent in a boring situation – like lectures What about dreaming while sleeping or burying one’s head a book Presence can be very strong without any sensor information  ”a self generated Virtual Reality” A book – a film from same book Maximum effect with hallucinogens

3 modes of presence 1. Really present, perceiving the existing environment 2. Tele or virtual presence  presence is transferred/generated by ”cheating” human senses 3. ”Mental” presence, presence is generated in the mind without (direct) sensor information