OFED (iWarp) Enhancements Felix Marti, Open Fabrics Alliance Workshop Sonoma, April 2008 Chelsio Communications

Verbs 2 Verbs should remain/become transport neutral  OFED does not support all iWarp Verbs; MIA are  Stag Alloc  Stag Dealloc  Fast Register Non-Shared Memory Region  RDMA Read Invalidate Local Stag  Invalidate Local Stag  User and Kernel space availability  Currently Bind Memory Window Kernel only?  APIs need to enable ‘fast path’ QP operations  Currently Bind MW not QP operation?`

Memory Handling Verbs  OFED consumers ask for better Memory Handling capabilities  i.e. NFSoverRDMA Memory Handling Dilemma: Table courtesy of Tom Talpey 3

iWarp Connection Setup 4  iWarp connections start life as TCP connections  Endpoints negotiate to upgrade to iWarp  Upgrade requires ‘initiator’ to send first iWarp PDU  Some (many) ULPs/Apps don’t adhere to the requirement – ahhh, fix the ULPs/Apps, pleeze

iWarp Connection Setup (2) OFED only solution – do we need standardization (IETF)? 5  Proposal (referred to as Peer 2 Peer support):  Negotiate p2p using MPA private data  MPA initiator advertises if it supports p2p  MPA responder advertises if it wants RTR and what type (OB RDMA Read, RDMA Write or Send)  MPA responder defers ‘initiating iWarp operations’ until RTR is received  MPA initiator sends RTR after successfully processing the MPA start response  Enhance RDMA-CM API to allow ULP/App to enable/disable p2p

Latest Chelsio iWarp Projects Chelsio is heavily investing into iWarp/OFED! 6  Peer 2 Peer Connection upgrade  OpenMPI (and the other MPIs)  RDS  NFSoverRDMA

Future Call To Action! Help Out! Maybe we can donate HW? 7  ULPs/Apps should migrate to RDMA-CM to take advantage of iWarp  i.e. ib_* tests shipping in OFED  As OFED becomes more transport neutral, life becomes easier  [4-tuple sharing with the host stack]

Thank You! 8