1. Inter-domain Socket Communications Supporting High Performance and Full Binary Compatibility on Xen March 5, 2008 Kangho Kim System Software Team Internet Server Group VEE ‘08

2. Outline  Motivation  Design & Implementation –Socket creation –Connect –Accept –Send/recv –Close –Live migration  Evaluation results  Conclusion

3. Motivation  Low bandwidth (Xen 3.0.3) We need XWAY!!

4. Motivation(cont’d)  Low latency (Xen 3.0.3)  Existing socket applications

5. Design & Implementation

6. Shared memory  Direct communication  TCP/IP bypass  Direct data copy  Shared memory Hardware VM Kernel User Domain 0Domain U XWAY Shared memory TCP

7. XWAY channel HeadTail SQ HeadTail RQ HeadTail RQ Event channel HeadTail SQ Shared memory for one XWAY channel Domain ADomain B

8. XWAY architecture socket application library VFS INET UNIXX25 … XWAY switch TCP IP XWAY protocol Network device driverXWAY driver BSD socket user kernel UDP

9. Dual channel  XWAY channel for data  TCP/IP channel for socket options and socket state socket application XWAY socket socket application XWAY socket TCP/IP channel XWAY channel

10. xway_sock Socket data structure for XWAY llseek: no_llseek poll: sock_poll open: sock_no_open release: sock_close readv: sock_readv writev: sock_writev sendpage: sock_sendpage release: inet_release bind: inet_bind connect: xsw_stream_connect accept: xsw_accept poll: xsw_poll listen: inet_listen shutdown: inet_shutdown sendmsg: inet_sendmsg recvmsg: sock_common_recvmsg sendpage: inet_sendpage close: xsw_close connect: xsw_v4_connect disconnect: xsw_disconnect accept: xsw_csk_accept ioctl: tcp_ioctl init: xsw_v4_init_sock shutdown: xsw_shutdown setsockopt: tcp_setsockopt getsockopt: tcp_getsockopt remote_dom_id eventchn_port eventchn_irq xring_send xring_recv xring_send_h xring_send_t xring_recv_h xring_recv_t … inode socket sock

11. Create Sock

12. Connect TCP channel Sock Connection helper XWAY channel SQ RQ SQ RQ

13. Accept TCP channel XWAY channel Sock XWAY channel

14. Send/Recv TCP channel Sock XWAY channel SQ RQ SQ RQ

15. Close Sock TCP channel XWAY channel SQ RQ SQ RQ XWAY channel

16. Poll TCP session XWAY session Sock

17. Live migration  Cut XWAY channels underneath XWAY socket –Redirect all data send/receive calls to TCP channels  Ready to live migration Hardware VM Kernel User Domain 0Domain U TCP/IP Hardware VM Domain 0

18. Live migration (cont’d) TCP channel Sock XWAY channel SQ RQ SQ RQ

19. Evaluation

20. Bandwidth  Netperf TCP_STREAM

21. Latency  Netperf (TCP_RR)

22. Application performance

23. DBT-1 performance

24. Execution time of API

25. Binary compatibility  Scp  Ssh  Apache – TCP_DEFER_ACCEPT  Vsftpd – connect, accept, MSG_PEEK  Proftpd – shutdown, connect, SO_LINGER  Telnet  MySQL  Netperf

26. Conclusion  XWAY design and implementation –Full binary compatibility for socket interface –High performance –Live migration support  Remaining work –UDP, … –Domain crash resiliency –Reducing CPU usage  Code release –April 20, 2007: xway ver 0.6 –June 4, 2007: xway ver 0.6.1 –By the end of March, 2008: xway ver 0.7 (live migration included)  Download –http://sourceforge.net/projects/xway

27. Questions or Comments  Kangho Kim –khk@etri.re.kr  Cheiyol Kim –gauri@etri.re.kr  Hyun-Sup Shin –superstarsup@etri.re.kr