1. OpenFabrics 2.0 rsockets+ requirements Sean Hefty - Intel Corporation Bob Russell, Patrick MacArthur - UNH

2. Data Streaming Current: RDMA CM for connection setup Single wait object and event queue –CM and CQ – use same fd –In-band disconnect notification Associate transport resource with an fd –fstat, dup2 Fork support –Migrate resources between user space and kernel chroot support www.openfabrics.org 2

3. Data Streaming Current: RDMA write with immediate Eliminate address and rkey exchange –Receiver selects key –Sender uses offset Eliminate need for immediate data –Generate event based on write: location and length www.openfabrics.org 3

4. Data Streaming Eliminate posting receives –No buffer is provided –Concern is overrunning CQ, not RQ Replace RDMA write with send –Receiver posts single buffer that hardware packs multiple messages into –Eliminates RDMA header Count of completed sends –Full completion data unnecessary www.openfabrics.org 4

5. Data Streaming Split received data into two buffers –Separate header and user data –Pack tightly, but use multiple buffers Partial completion event –Notification of partial transfer for large requests –Allow receive side to being processing www.openfabrics.org 5

6. Data Streaming Nonblocking support –Signal when transport is ready to accept new data –Available QP and CQ resources, send credits Keepalive support –0-byte send that does not generate a remote event –Similar to RDMA-write, but eliminate header www.openfabrics.org 6

7. Datagram User selectable transport address (QPN) –High QPN lookup costs –Message backlog Multi-receive message buffer –Single buffer receives multiple messages –Split received data into two buffers Separate header and user data Pack tightly, but use multiple buffers www.openfabrics.org 7

8. Datagram Fast address resolution –Compact address data Multicast support –Fast access to multicast group www.openfabrics.org 8

9. General Requests Increase size of immediate data –Provide easy mechanism to discover if immediate data is supported and size Slab based allocation for receive buffers –Eliminate wasted space dealing with max message size Eliminate posting of ‘dummy’ receive for immediate data www.openfabrics.org 9

10. General Requests Add timeout parameters to all CM operations –E.g. connect, accept, disconnect, join multicast Timeout parameters for reading events Ability to cancel a pending I/O –Including CM operations www.openfabrics.org 10

11. General Requests Error handling must be consistent –Do not leave to providers –Document which error codes every call can return Similar to POSIX error code documentation –Use a single error return convention Return -1 and set errno? Return –errno? (prefered) Return +errno? –Consistent error values in events Do not mix transport and errno values –Easy mechanism to display error text www.openfabrics.org 11

12. General Requests Query current status of local queues –Generating an async event (e.g. SRQ) compounds the issue of dealing with multiple fd’s –Eliminate need for these events or provide in-band notification Support memory registration across multiple devices –Register at the system level, not per PD per HCA www.openfabrics.org 12

13. General Requests Need simple, programmatic way to detect memory alignment restrictions –Or avoid any alignment needs Need better way to discover supported ‘inline’ sizes –Providers should ensure that that reported values actually improve performance www.openfabrics.org 13

14. General Requests Define reasonable minimum requirements on providers for: –Number of SGEs –Inline size –Immediate data size –CM private data length With a supported minimum for any message www.openfabrics.org 14

15. General Requests Asynchronous interface can be source of races –E.g. completions before call returns –Have provider update user counters before generating completion Support multiple providers at run-time Provide test suite to verify provider conformance to API specifications –Example programs –Error conventions –Min/max values www.openfabrics.org 15