OpenFabrics 2.0 or libibverbs 1.0 Sean Hefty Intel Corporation

Pre-OpenIB Multiple IB vendors each with a proprietary software stack –Multiple versions of the ‘verbs’ interface –Binary incompatible Formation of OpenIB –Focused on creating a single, open source software stack for InfiniBand –Apps have 1 API to target, distros have 1 solution to support –Greatly increased adoption 2

OpenIB Resulting software stack was hardware facing –Verbs is a hardware semantic –Was never intended to be an API –Application developers have always disliked it OpenIB expanded to include iWarp –Changed name to OpenFabrics –Folded iWarp support under InfiniBand API 3

OpenFabrics Today The fundamental stack is largely unchanged The result: –There are multiple, binary incompatible versions of the ‘verbs’ interface –There are vendor-specific APIs FCA, MXM, PSM, UCCS Sounds a lot like the pre-OpenIB days 4

Why Not Just Extend Verbs? Author’s prediction: will not succeed –OIB created a single verbs solution, which later split The split was a direct result of needing extensions –Additional APIs have been already been developed that are NOT part of verbs or extended verbs Existence proof against –There are fundamental issues with the API that will prohibit scaling to very large systems 5

Proposed Alternative Develop application facing APIs and let each vendor determine the best way to support those APIs –There is no intent to disadvantage any hardware solution Define mechanisms for migrating providers and apps to the new framework 6

(Scalable) Fabric Interfaces Q: What is implied by incorporating interface sets under a single framework? Objects exist that are usable between the interfaces Isolated interfaces turn the framework into a complex dlopen Interfaces are composable May be used together 7 Fabric Interfaces Message Queue Control Interface Control Interface RDMA Atomics Active Messaging Tag Matching Collective Operations CM Services Cohesive interfaces, not merely a union

Migrating Providers from Verbs to FI libfabric CM Services libibverbs FI RDMA CM Verbs Provider Message Queue RDMA Verbs Verbs Provider ibverbs abstraction layer Providers can integrate into framework with minimal changes Providers leverage abstraction layer and RDMA CM capabilities Add optimize data transfers and completion processing calls to avoid translation overhead 8

Migrating Apps from Verbs to FI Expose ‘verbs’ interfaces directly from FI Use macros to convert ‘libibverbs’ exported calls to FI calls Or layer libibverbs over libfabric Applications must recompile Minimal benefit to app 9

Migrating Apps from Verbs to FI Define ‘verbs’ compatibility mode –Allows mapping objects between interfaces –E.g. QP  fabric socket CQ  EC –Restricts implementation –Mapping must be documented Allow software to adopt new interfaces selectively –E.g. send/recv/ec_read libfabric Dual-Provider Library verbs libibverbs SFI SFI Provider Verbs Provider 10