A DISTRIBUTED FRAMEWORK FOR RELAYING STEREO VISION FOR TELEROBOTICS M. Al-Mouhamed, O. Toker, A. Iqbal, and M. Nazeeruddin

2 Contents Introduction Distributed framework Video server Video client Server optimization Augmented reality (AR) AR support Telerobotic client framework Client-server framework performance evaluation Conclusion

3 Introduction Telerobotics: humans to extend their manipulative skills over a distance, remote manual work,.. Telerobotic applications  Scaled-down: nano-scale, micro-surgery, surgery, clean-room  Hazardous: nuclear decommissioning & inspection, disposal of dangerous objects, minefield clearance, operation in harsh environments like in space, underwater, ice, desert,..  Safety: rescue, fire fighting,..  Security: surveillance, reconnaissance,..  Unmanned: oil platform inspection, repair,..  Teaching, training, entertainment,..

4 Introduction … (cont.) Challenge: extending eye-hand motion coordination through a network with high-quality perception, dexterity, and intelligent computer aided teleoperation.  In natural eye-hand motion coordination, operator sees his hand and reacts accordingly.  Telerobotics: Operator holds a master arm to dictate his hand motion, Motion is transmitted to a remote slave arm and reproduced (replica), Operator wears a head-mounted display (HMD) to see in 3D the effects of his motion on the remote tool, Operator does not see his hand (HMD) nor the master arm, his hand is logically mapped to the remote tool, Operator logically acts on the remote tool seen through the HMD.  Stereo vision: 3D perception of remote scene, a metric to calculate 3D position and orientation of objects, a tool to augment the real space (augmented reality),..

5  A reliable and portable client-server frame work for Computer Aided Telerobotics (CAT)  Real-time streaming of video data from server to client  Multithreaded system using Visual C#, DirectX, and Windows sockets for efficient capturing and relaying of video  DirectX, COM interfaces for graphic functionalities like DirectShow  Client support to AR, e.g. graphic modeling, superimposing animated graphics, etc.  Support CAT operability, e.g. tool definition, activation of server automation tasks, indexing function, space scalability, shared control, impedance control, etc. Distributed framework

6 Development of a client-server framework for grabbing and relaying of stereo video Video server:  DirectShow for effective interfacing, capturing, and playback of video data  Capture or grab stereo images from two digital cameras (FireWire interface)  Establish a reliable client-server connection  Upon requests from the client send the stereo frames (two pictures) to the client through windows sockets  Camera control, disparity, focus, orientation, etc

7 Distributed framework

8 Video client:  Detect and establish the connection with server  Establish a fast graphic display system to show the pictures received from the server  Display the pictures arrived from the server and continue in a loop each time asking a new stereo frame from the server  Allow the viewer to adjust the alignment of pictures on the HMD to compensate for the misalignment and non-linearity present in the camera at server  Control server camera setup

9 Distributed framework (Video client)  Augmented reality: superimpose graphics (virtual) over the stereo views of real scene to augment the stereo space  Avoid stop-and-wait in delayed teleoperation  At client, emulate a sequence of simple actions over current state of server robot using a model of environment with interactions  Send a converging solution for execution at remote robot  Supervise real execution while receiving feedback  Seamlessly adds virtual (registered) objects into the real scene to avoid any perplexities for the teleoperator  Carry out task planning (non-invasive)  Pre-run automatic tasks (tool switching, imp. control,..)

10 Distributed framework

11 Video server Network services like send/receive for video/data using windows sockets Stereo video setup uses synchronous windows sockets SampleGrabber (DirectShow) to capture video frames coming through a stream from a stereo camera setup Two schemes to transfer the video data, usage of multiple threads, optimization to reduce network traffic General overview of the image grabbing and displaying system

12 Video server

13 Video client GDI (Graphics Device Interface) is used to display the received pictures from the server

14 Client framework

15 …………….. Server optimization (single buffer, serialized transfer) Internet internetinternet  …………..

16 Server optimization (double buffer, concurrent transfer) Optimize the transfer of video data using thread engineering on the server Concurrent thread execution among capture and transfer threads is achieved using double buffers on the server side The thread responsible for sending the video data over the network will not wait after receiving a request for a picture from the client Limited overhead in the allocation of extra buffer in the server DRAM and management Pipelined execution: higher stereo frame rate due to time overlap between capture thread and network transfer thread Need independent hardware for capture and transfer paths

17 Server optimization (double buffer, concurrent transfer) ………..……….. Internet

18 Augmented reality (AR) Camera model  A one-to-one mapping of scene and graphics  Weak-perspective camera model  Affine invariant projection for 2D-3D mapping  Identify two projection matrices (left and right) Microsoft DirectX  APIs for developing 2D and 3D graphics  Graphics drawn on a DirectX surface (access to pixels)  HAL provides a set of graphics functions on hardware devices  Primary surface is the current buffer, next data frame to off-screen secondary surface, and use of page flipping Server:  Setting up cameras  Placement and removal of reference frame Client:  StereoSocketClient component  IdentifyCamera component  RobotModel component  DXInterface component

19 AR support

20 Telerobotic client framework

21 Video Client-Server Framework Performance Evaluation Testing on University campus network Client-server P-4, 2 GHz, 100 Mbps NIC A video picture is 288x360 pixels, a pixel is 3 bytes, a picture is 304 KB, a stereo is 0.6 MB, a stereo frame needs 5 Mbps bandwidth Metrics: quality of depth perception, time delays, etc. Video clips on teleoperation tasks like peg-in-hole insertion, assembly of a small water pump, wire-wrapping, pouring water,..

22 Video Client-Server Framework Performance Evaluation Copying from SampleGrabber to DRAM Case 1.1:  A single thread to copy stereo frame from SampleGrabber to DRAM on server 300 stereo frames Mean value = ms 95% CI between ms and ms.

23 Video Client-Server Framework Performance Evaluation Copying from SampleGrabber to DRAM Case 1.2:  First thread to copy stereo frame from SampleGrabber to DRAM on server  Second thread to read a force data without transfer 300 stereo frames Mean value = ms 95% between 8 ms and 150 ms.

24 Video Client-Server Framework Performance Evaluation Copying from SampleGrabber to DRAM Case 1.3:  First thread to copy stereo frame from SampleGrabber to DRAM on server  Second thread to read a force data with transfer  300 stereo frames  Mean value is ms  9.43 ms additional for adding network transport thread.  Need thread management

25 Video Client-Server Framework Performance Evaluation

26 Video Client-Server Framework Performance Evaluation A magnified plot of inter-arrival times of force in the presence of stereo video and command streams.

27 Video Client-Server Framework Performance Evaluation Transferring over the campus network Case 2.1:  Single video thread  Single buffer  Serialized transfer  300 stereo frames  Mean value = 86.1 ms  stereo frames/second.

28 Video Client-Server Framework Performance Evaluation Transferring over the campus network Case 2.2:  Single video thread  Double buffer  Concurrent transfer  60,000 stereo frames  Mean value = ms  17 stereo frames/second.  90% CI between 56.0 and 64.8 ms.

29 Video Client-Server Framework Summary SchemeCameras to server DRAM (ms) Server to client (ms) Frames per second Single buffer, serialized Double buffer, concurrent uncompressed stereo fps of size 360X288 pixels

30 Conclusion A stereo video client-server framework using Multi-threaded programming, Visual C++ and Visual C#.NET programming languages Reliable connection between client and server Visualization support to eye-shuttering and HMD page flipping A mathematical framework to support generation of complex geometrical shapes and interactive AR AR teleoperation tools to overcome network delays and reduce needed bandwidth A stereo video transfer rate of 18 fps over University campus network Excellent depth perception using HMD techniques even for scaled- down operations

31