SC’13 BoF Discussion Sean Hefty Intel Corporation

SC’13 BoF Feedback Most people I’ve spoken with are highly supportive of the concepts being discussed –General alignment Active discussion at SC ‘13 –Engagement is high Good participation Want even wider focus outside of MPI –Diverse viewpoints helping to drive conversation More likely to produce better product Need to understand all concerns Address as many concerns as reasonable 2

Main Concerns from BoF Application centric APIs may require new protocols –Must ensure wire compatibility RDMA CM provides a simpler abstraction over verbs –How does this effort differ Some applications want low-level access –Technology specific apps 3

Evolving OFA Software from the Viewpoint of libibverbs Analyze libibverbs and librdmacm details –Selected issues –Previously covered fast path call overhead Identify specific areas for improvement Abstract the improvements to create a higher- performing, more extensible framework 4

Current libraries 5 libibverbs Verbs Provider Infiniband verbs interface User to kernel ‘command’ interface librdmacm Verb Helpers

libibverbs 6 libibverbs Verbs Provider Infiniband verbs interface User to kernel ‘command’ interface 2 interfaces are exported – one for applications and one for providers Application interface is based on IB defined hardware interface Fast path calls go directly to provider (inline - not exported) Data structures expose internal implementation details

libibverbs Structures struct ibv_qp –pthread objects (mutex, cond) –internal counter –internal handle –qp data (state, qpn, type) –5 data pointers –user context struct ibv_cq – pthread objects (mutex, cond) – internal counters – internal handle – 2 data pointers – user context 7 Exposes internal implementation IB specific QP data -Not useful for iWarp -Not needed with rdma cm IB specific QP data -Not useful for iWarp -Not needed with rdma cm Data pointers may not be valid or needed by app

libibverbs Structures struct ibv_context –struct ibv_device *device –struct ibv_context_ops ops; –int cmd_fd; –… 8 Provider implemented functions Embedded structure – fixed size! -Adding new calls is non-trivial -No mechanism to do so upstream Embedded structure – fixed size! -Adding new calls is non-trivial -No mechanism to do so upstream Provider functions associated with an opened device -Cannot optimize call based on use -Branches in code path and larger data structures Provider functions associated with an opened device -Cannot optimize call based on use -Branches in code path and larger data structures

librdmacm Original APIs –Create ID, resolve address, resolve route –Connect, listen, accept, reject Newer APIs –Getaddrinfo, create endpoint Verb helpers –Register memory –Send/receive, rdma read/write, UD send 9 No-op for iwarp and RoCE Scalability issues Required for UD Addresses scalability and ease of use Simplified calls, but limited by verbs API

Evolving ‘Verbs’ Merge libibverbs and librdmacm Eliminate duplicate functionality Remove internal implementation details from the API –Abstract data structures Associate functionality with specific objects –Enable optimized code paths Abstract objects for extensibility –Enable other use cases 10 These are goals of the proposed framework

BoF Concern Application centric APIs may require new protocols –Must ensure wire compatibility Transport protocol is exposed by framework API definitions and protocol definitions are separate Vendor-specific protocols are already being defined outside of verbs –PSM, MXM, FCA TBD: discuss related protocols –Address resolution, route lookup, connection setup 11

BoF Concern RDMA CM provides a simpler abstraction over verbs –How does this effort differ RDMA CM abstractions are limited by functions exported by verbs –Still filter to single post send/recv call RDMA CM is highly dependent on, but maintained separately –RDMA CM use is required for iWarp RDMA CM functionality is carried forward 12

BoF Concern Some applications want low-level access –Technology specific apps Low-level access will still be possible Focus is on ease of use (librdmacm model) with high-performance (libibverbs fast path) 13

(Scalable) Fabric Interfaces Q: What is implied by incorporating interface sets under a single framework? Objects exist that are usable between the interfaces Isolated interfaces turn the framework into a complex dlopen Interfaces are composable May be used together 14 Fabric Interfaces Message Queue Control Interface Control Interface RDMA Atomics Active Messaging Tag Matching Collective Operations CM Services Cohesive interfaces, not merely a union

Migrating Providers from Verbs to FI libfabric CM Services libibverbs FI RDMA CM Verbs Provider Message Queue RDMA Verbs Verbs Provider ibverbs abstraction layer Providers can integrate into framework with minimal changes Providers leverage abstraction layer and RDMA CM capabilities Add optimize data transfers and completion processing calls to avoid translation overhead 15

Migrating Apps from Verbs to FI Expose ‘verbs’ interfaces directly from FI Use macros to convert ‘libibverbs’ exported calls to FI calls Or layer libibverbs over libfabric Applications must recompile Minimal benefit to app Dependent on accessed data structures and interfaces 16

Migrating Apps from Verbs to FI Define ‘verbs’ compatibility mode –Allows mapping objects between interfaces –E.g. QP  fabric socket CQ  EC –Restricts implementation –Mapping must be documented Identified by protocol Allow software to adopt new interfaces selectively –E.g. send/recv/ec_read libfabric Dual-Provider Library verbs libibverbs FI FI Provider Verbs Provider 17