Middleware Support for RDMA-based Data Transfer in Cloud Computing Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, Thomas Robertazzi Department of Electrical and Computer Engineering Stony Brook University

Outline n Introduction and Background n Middleware Design and RFTP application n Experimental Results n Conclusion

Outline n Introduction and Background l Overview l RDMA Semantics n Middleware Design and RFTP application n Experimental Results n Conclusion

Today’s Data-intensive Applications n Explosion of data, and massive data processing n Scalable storage systems n Ultra-high speed network for data transfer: 40/100Gbps networks n Reliable Transfer (error checking and recovery) at 40/100G speed, burden on processing power

ANI Ultra-high Speed Network

End-to-End 40/100G Networking 100G APPS FTP /100G NIC 40/100 Gbps Backbone 40/100 Gbps Backbone 100 G APPS FTP /100G NIC End-to-End Networking at 40/100 Gbits/s Our project and its role

Protocol Offload and Hardware Acceleration n TCP/IP Offload Engine (TOE) n Protocol Offload Engine (POE) n Remote Directory Memory Access (RDMA) l Kernel by pass l Zero-copy

Applications over different RDMA implementations

RDMA Semantics n Channel Semantic – SEND/RECV l Two-side operation l Both data source and data sink are involved. The sink pre- posts a list of buffers into receive queue. n Memory Semantic – RDMA WRITE/RDMA READ l One-side operation l Credit-based. The sink advertises its available registered memory to the source for RDMA_WRITE operation. n We use RDMA WRITE operation to deliver user payload(128KB ~ 4MB per block), while use SEND/RECV to exchange control messages( ~2KB).

Outline n Introduction and Backgroud n Middleware Design and RFTP application l Middleware Layer l Middleware Software Architecture l Asynchronous Communication Events design l RFTP Modules l RDMA extension to standard FTP protocol n Experimental Results n Conclusion

Middleware Layer InfiniBandRoCEiWARP IB Verbs libibverbs RDMA CM librdmacm Application Buffer Management Connection Management Event Dispatch/Join Task Scheduling Middleware OFED Hardware

Middleware – Multi-threaded Architecture ThreadsData Structure CQ QP-1QP-2QP-n Data Block List Receive Control Message List Send Control Message List Remote MR Info List application system Queue Pair List Memory Sender CE dispatcher CE slave-n... CE slave-2 CE slave-1 Logger Hardware HCA

Communication Events n Session ID negotiation l Each data transfer task will be assigned a unique session ID n Number of data connection negotiation l Establish several parallel connections n Memory region credit request and response l The source issues request of Memory regions’ information l The sink feedbacks several credit according to buffer status n Block completion notification l The source issues a notification to the sink which block’s data is ready

Parallel and Pipelined Data Transfer n Explore parallelism of RDMA operations l Multiple active data streams l Each stream uses a pipelined execution n Out-of-order blocks l Reorder l Deliver in-order blocks to application

RDMA-enabled FTP - RFTP RDMA Middleware FTP … Disk I/O Module InfiniBandiWARPRoCE Verbs Communication manager SSDMagnetic Disk Driver API Hardware Operating System Middleware Application Buffer Manage I/O Scheduling Connection Manage Event Dispatch Task Scheduling Direct I/O API

RDMA extension to standard FTP protocol

Outline n Introduction and Backgroud n Middleware Design and RFTP application n Experimental Results l Testbed Setup l LAN results l MAN results n Conclusion

Testbed Setup - LAN 10Gbps 40Gbps

Testbed Setup - MAN 40Gbps RoCE link RTT = 3.6ms

LAN – Bandwidth and CPU Usage Comparison

MAN – RFTP evaluation

Outline n Introduction and Background n Middleware Design and RFTP application n Experimental Results n Conclusion

Conclusion n Data-intensive application in cloud computing require efficient data transfer protocols to fully utilize the capacity of advanced network infrastructure n Designed and implemented a RDMA-based middleware layer n Developed a FTP application based on this middleware layer n Tested the performance of our design and implementation on both LAN and long-haul MAN links

Thank you