1. Scalable RDMA Software Solution Sean Hefty Intel Corporation

2. Overview Common Addressing ModelCommon Addressing Model Communication ManagementCommunication Management Device HandlingDevice Handling Infiniband Software Solution Path ResolutionPath Resolution Multicast SupportMulticast Support RDMA Software Solution Infiniband Scalability

3. Common Addressing Model Simplify user interfaceSimplify user interface Permit use of standard name services and interfacesPermit use of standard name services and interfaces –Socket addresses, name service resolution Consistent addressing model across RDMA devices

4. Address Mapping Service RDMA addresses Map IP address to RDMA device Generic workqueue

5. Communication Management Socket like semanticsSocket like semantics IP network addressesIP network addresses RDMA port spacesRDMA port spaces –Conceptual port space sharing –TCP, UDP, SDP, SCTP Common connection interface for all RDMA devices

6. RDMA CM Transport independent interface Acquire device before connecting Wildcard listens across all devices Resolve routing before connecting Handle device hotplug events

7. RDMA CM Optionally bind to a specific device Bind to local device New connection Serialize removal with connect events Select fabric path

8. Infiniband Scalability Support scale-out to thousands of nodesSupport scale-out to thousands of nodes Efficient MPI collective operationsEfficient MPI collective operations Prevent SA stormsPrevent SA storms Reduce hardware requirementsReduce hardware requirements –High performance UD –Multicast endpoints –Minimal memory footprint

9. Path Resolution Reduce connection setup timeReduce connection setup time Decrease SA flood on app startupDecrease SA flood on app startup Application selected routesApplication selected routes –I view MPI as an app MultiPath record supportMultiPath record support –Path independence

10. Path Resolution Efficient SA interaction Radix tree – w/ variable sized key Still requires scalable SA Path records

11. Multicast Support Creation attributes outside of specCreation attributes outside of spec SA tracks join/leave requests per portSA tracks join/leave requests per port Requires local reference countingRequires local reference counting Serialize operations to SASerialize operations to SA Queue join/leave requestsQueue join/leave requests Architectural support issues

12. Multicast Support Multicast group identified by MGID Map IP address to MGID Support send-only and full membership Creation? –Automatic using IPoIB for attributes –Separate application

13. Current Status RDMA CMRDMA CM –Submitted for 2.6.17 –Merging iWarp support Path resolutionPath resolution –Verify implementation –Enable route selection algorithms MulticastMulticast –Next at bat

14. Future Work VerbsVerbs –QP redirection MADMAD –Large transfers –Dual-sided RMPP Subnet administrationSubnet administration –Where to start?

15. Future Work RDMA CMRDMA CM –Failover – multiple routes –UD support –IPv6 Path resolutionPath resolution –Scalable memory footprint –Reduce network requirements –Efficient change detection