Inter-domain Socket Communications Supporting High Performance and Full Binary Compatibility on Xen March 5, 2008 Kangho Kim System Software Team Internet Server Group VEE ‘08

Outline  Motivation  Design & Implementation –Socket creation –Connect –Accept –Send/recv –Close –Live migration  Evaluation results  Conclusion

Motivation  Low bandwidth (Xen 3.0.3) We need XWAY!!

Motivation(cont’d)  Low latency (Xen 3.0.3)  Existing socket applications

Design & Implementation

Shared memory  Direct communication  TCP/IP bypass  Direct data copy  Shared memory Hardware VM Kernel User Domain 0Domain U XWAY Shared memory TCP

XWAY channel HeadTail SQ HeadTail RQ HeadTail RQ Event channel HeadTail SQ Shared memory for one XWAY channel Domain ADomain B

XWAY architecture socket application library VFS INET UNIXX25 … XWAY switch TCP IP XWAY protocol Network device driverXWAY driver BSD socket user kernel UDP

Dual channel  XWAY channel for data  TCP/IP channel for socket options and socket state socket application XWAY socket socket application XWAY socket TCP/IP channel XWAY channel

xway_sock Socket data structure for XWAY llseek: no_llseek poll: sock_poll open: sock_no_open release: sock_close readv: sock_readv writev: sock_writev sendpage: sock_sendpage release: inet_release bind: inet_bind connect: xsw_stream_connect accept: xsw_accept poll: xsw_poll listen: inet_listen shutdown: inet_shutdown sendmsg: inet_sendmsg recvmsg: sock_common_recvmsg sendpage: inet_sendpage close: xsw_close connect: xsw_v4_connect disconnect: xsw_disconnect accept: xsw_csk_accept ioctl: tcp_ioctl init: xsw_v4_init_sock shutdown: xsw_shutdown setsockopt: tcp_setsockopt getsockopt: tcp_getsockopt remote_dom_id eventchn_port eventchn_irq xring_send xring_recv xring_send_h xring_send_t xring_recv_h xring_recv_t … inode socket sock

Create Sock

Connect TCP channel Sock Connection helper XWAY channel SQ RQ SQ RQ

Accept TCP channel XWAY channel Sock XWAY channel

Send/Recv TCP channel Sock XWAY channel SQ RQ SQ RQ

Close Sock TCP channel XWAY channel SQ RQ SQ RQ XWAY channel

Poll TCP session XWAY session Sock

Live migration  Cut XWAY channels underneath XWAY socket –Redirect all data send/receive calls to TCP channels  Ready to live migration Hardware VM Kernel User Domain 0Domain U TCP/IP Hardware VM Domain 0

Live migration (cont’d) TCP channel Sock XWAY channel SQ RQ SQ RQ

Evaluation

Bandwidth  Netperf TCP_STREAM

Latency  Netperf (TCP_RR)

Application performance

DBT-1 performance

Execution time of API

Binary compatibility  Scp  Ssh  Apache – TCP_DEFER_ACCEPT  Vsftpd – connect, accept, MSG_PEEK  Proftpd – shutdown, connect, SO_LINGER  Telnet  MySQL  Netperf

Conclusion  XWAY design and implementation –Full binary compatibility for socket interface –High performance –Live migration support  Remaining work –UDP, … –Domain crash resiliency –Reducing CPU usage  Code release –April 20, 2007: xway ver 0.6 –June 4, 2007: xway ver –By the end of March, 2008: xway ver 0.7 (live migration included)  Download –

Questions or Comments  Kangho Kim  Cheiyol Kim  Hyun-Sup Shin