1. Early Experiences with NFS over RDMA OpenFabric Workshop San Francisco, September 25, 2006 Sandia National Laboratories, CA Helen Y. Chen, Dov Cohen, Joe Kenny Jeff Decker, and Noah Fischer hycsw,idcoehn,jcdecke,nfische@sandia.gov SAND 2006-4293C

2. 2 Outline Motivation RDMA technologies NFS over RDMA Testbed hardware and software Preliminary results and analysis Conclusion Ongoing work and Future Plans

3. 3 What is NFS A network attached storage file access protocol layered on RPC, typically carried over UDP/TCP over IP Allow files to be shared among multiple clients across LAN and WAN Standard, stable and mature protocol adopted for cluster platform

4. 4 NFS Scalability Concerns in Large Clusters Large number of concurrent requests from parallel applications Parallel I/O requests serialized by NFS to a large extend Need RDMA and pNFS NFS Server Application 1 Application 2 Application N Concurrent I/O

5. 5 How DMA Works

6. 6 How RDMA Works

7. 7 Why NFS over RDMA NFS moves big chunks of data incurring many copies with each RPC Cluster Computing –High bandwidth and low latency RDMA –Offload protocol processing –Offload host memory I/O bus –A must for 10/20 Gbps networks http://www.ietf.org/internet-drafts/draft-ietf-nfsv4- nfs-rdma-problem-statement-04.txt

8. 8 The NFS RDMA Architecture NFS is a family of protocol layered over RPC XDR encodes RPC requests and results onto RPC transports NFS RDMA is implemented as a new RPC transport mechanism Selection of transport is an NFS mount option NFS v2 NFS v3 NFS v4 NLM NFS ACL RPC XDR UDPTCPRDMA Brent Callaghan, Theresa Lingutla-Raj, Alex Chiu, Peter Staubach, Omer Asad, “NFS over RDMA”, ACM SIGCOMM 2003 Workshops, August 25-27, 2003

9. 9 This Study

10. 10 OpenFabrics Software Stack Offers a common, open source, and open development RDMA application programming interface http://openfabrics.org/http://openfabrics.org/

11. 11 Testbed Key Hardware Mainboard: Tyan Thunder K8WE (S2895) http://www.tyan.com/products/html/thunderk8we.html –CPU – Dual 2.2 Ghz AMD Opteron Skt940 http://www.amd.com/us- en/assets/content_type/white_papers_and_tech_docs –Memory – 8 GB ATP 1GB PC3200 DDR SDRAM on NFS server and 2 GB CORSAIR CM725D512RLP- 3200/M on client IB Switch: Flextronics InfiniScale III 24-port switch http://mellanox.com/products/switch_silicon.php IB HCA: Mellanox MT25208 InfiniHost III Ex http://www.mellanox.com/products/shared/Infinihostglossy.pdf

12. 12 Testbed Key Software Kernel: Linux 2.6.16.5 with deadline I/O scheduler NFS/RDMA release candidate 4 – http://sourceforge.net/projects/nfs-rdma/ oneSIS used to boot all the nodes http://www.oneSIS.org OpenFabric IB stack svn 7442 http://openib.org

13. 13 Testbed Configuration One NFS server and up to four clients –NFS/TCP vs. NFS/RDMA –IPoIB and IB RDMA running SDR Ext2 with Software RAID0 backend Clients ran IOZONE writing and reading 64KB records and 5GB aggregate file size used –To eliminate cache effect on client –To maintain consistent disk I/O on server Allowing the evaluation of NFS/RDMA transport without being constrained by disk I/O System resources monitored using vmstat at 2s intervals Server Clients IB switch

14. 14 Local, NFS, and NFS/RDMA Throughput LocalNFS (IPoIB)NFS/RDMA Write (MB/s)266.11100.02130.26 Read (MB/s)1518.20179.94692.94 Reads are from server cache reflecting –TCP RPC transport achieved ~180 MB/s (1.4 Gb/s) of throughput –RDMA RPC transport was capable of delivering ~700MB/s (5.6Gb/s) throughput RPCNFSDCOUNT=8 /proc/sys/sunrpc/svc_rdma/max_requests=16

15. 15 NFS & NFS/RDMA Server Disk I/O Writes incurred disk I/O issued according to deadline scheduler –NFS/RDMA server has higher incoming data rate, and thus higher block I/O output rate to disk –NFS/RDMA data-rate bottlenecked by the storage I/O rate as indicated by the higher IOWAIT time

16. 16 NFS vs. NFS/RDMA Client Interrupt and Context Switch NFS/RDMA incurred ~1/8 of Interrupts, completed in a little more than 1/2 of the time NFS/RDMA showed higher context-switch rates indicating faster processing of application requests Higher throughput comparing to NFS!

17. 17 Client CPU Efficiency CPU per MB of transfer:  t )    cpu/100 / file-size Write NFS 0.00375 NFS/RDMA = 0.00144 61.86% more efficient! Read NFS = 0.00435 NFS/RDMA = 0.00107 75.47% more efficient! Improved application performance

18. 18 Server CPU Efficiency CPU per MB of transfer:  t)    cpu/100 / file-size Write NFS = 0.00564 NFS/RDMA = 0.00180 68.10% more efficient! Read NFS = 0.00362 NFS/RDMA = 0.00055 84.70% more efficient! Improved system performance

19. 19 To minimize the impact of disk I/O –One 5GB, two 2.5GB, three 1.67GB, four 1.25GB Ignored rewrite and reread due to client-side cache effect Scalability Test - Throughput

20. 20 Scalability Test – Server I/O NFS RDMA transport demonstrated faster processing of concurrent RPC I/O requests and responses from and to the 4 clients than NFS Concurrent NFS/RDMA writes were impacted more by our slow storage as indicated by the close to 80% CPU IOWAIT times

21. 21 Scalability Test – Server CPU NFS/RDMA incurred ~½ the CPU overhead and for half of the duration, but delivered 4 times the aggregate throughput comparing to NFS NFS/RDMA write-performance was impacted more by the backend storage than NFS, as indicated by the ~70% vs. ~30% idle CPU time waiting for IO to complete

22. 22 Compared to NFS, NFS/RDMA demonstrated: –impressive CPU efficiency –and promising scalability NFS/RDMA will Improve application and system level performance! NFS/RDMA can easily take advantage of the bandwidth in 10/20 Gigabit network for large file accesses Preliminary Conclusion

23. 23 SC06 participation –HPC Storage Challenge Finalist Micro benchmark MPI Applications with POSIX and/or MPI I/O –Xnet NFS/RDMA demo over IB and iWARP Ongoing Work

24. 24 Future Plans Initiate study of NFSv4 pNFS performance with RDMA storage –Blocks (SRP, iSER) –File (NFSv4/RDMA) –Object (iSCSI-OSD)?

25. 25 NFSv3 –Use of ancillary Network Lock Manager (NLM) protocol adds complexity and limits scalability in parallel I/O –No attribute caching requirement squelches performance NFSv4 –Use of Integrated lock management allows byte range locking required for Parallel I/O –Compound operations improves efficiency of data movement and … Why NFSv4

26. 26 Why Parallel NFS (pNFS) pNFS extends NFSv4 –Minimum extension to allow out-of-band I/O –Standards-based scalable I/O solution Asymmetric, out-of-band solutions offer scalability –Control path (open/close) different from Data Path (read/write) http://www3.ietf.org/proceedings/04nov/slides/nfsv4-8/pnfs-reqs-ietf61.ppt

27. 27 Acknowledgement The authors would like to thank the following for their technical input –Tom Talpey and James Lentini from NetApp –Tom Tucker from Open Grid Computing –James Ting from Mellanox –Matt Leininger and Mitch Sukalski from Sandia