OFED - Status and Process November 2007 Tziporet Koren

2 Mellanox Technologies Agenda  OFED 1.3 Status  OFED Development Process  OFED Future Discussion

3 Mellanox Technologies OpenFabrics Software Stack

4 Mellanox Technologies OFED – Update Since Sonoma  OFED 1.2 was released on 22-Jun  Kernel base:  OFED was released on 16-Aug  Kernel base  ConnectX support  OFED was released on 14-Nov  RDS in GA for Oracle 11g RAC  OFED 1.3 – in middle of release phase  Kernel base

5 Mellanox Technologies OFED 1.3 Components  HCA/NIC Drivers  Mellanox, QLogic, IBM, Chelsio, NetEffect  Core: verbs, mad, SMA, CMA, SA cache  IPoIB  SDP  SRP, iSER  SRP Target  RDS  Qlogic_VNIC  UDAPL  OSM  Diagnostic tools  NFS-RDMA – still open  Bonding module  MPI Components  MVAPICH  Open MPI  MVAPICH2  Benchmark tests  Proprietary MPIs: Intel, HP, Scali  Proprietary SMs: Cisco, Voltaire, Qlogic OFA developmentAdd on Tested with

6 Mellanox Technologies Main New Features - General  Base kernel:  Install  Minimize integration effort into OS distribution  User level split into separately-installable RPMs  Based on feedback from Novell and Redhat  Package  Sources arrangement for the end user  Added the make-dist script  Reduce compilation warnings of backports

7 Mellanox Technologies General Features – Cont.  QoS  OSM, CM & CMA, ULPs: SRP, SDP, iSER, IPoIB  Updated SA cache  Integrated with sa_query module  Use of SA events (GID up/down) to provide additional synchronization  User-space control for dynamic enable/disable and update  Extended RC (XRC) Support  New verbs to enable XRC  New modules  NES (NetEffect driver)  SRP Target: Based on Mellanox IBGD SRPT and interfaces with SCST scsi target middle level driver

8 Mellanox Technologies Main New Features – ULPs  IPoIB  NAPI  Stateless offloads – for Datagram mode  Small message performance improvements  RDS: RDMA with Zero copy (API version 3)  SDP:  Asynch IO  Send Zero Copy  uDAPL 2.0  In addition to uDAPL 1.0

9 Mellanox Technologies Main New Features – Management  OpenSM  Quality of Service manager  QoS implementation of the IBTA annex  QoS policy parser and support for SA (PR, MPR)  Scalability/stability improvements  Performance improvements of the routing algorithms (min-hops, up/down, lash)  Optimized port and switch tables update policy  “Native” daemon mode  Performance manager  Fetches and stores performance counters  OpenSM integrated, work in any OpenSM state (standby, disabled)  Plug-in interface  Diagnostic tools enhancements  IBSIM – IB fabric simulator (new)  Emulates user_mad layer  useful for OpenSM and management tools testing

10 Mellanox Technologies Main New Features - MPI  MVAPICH 1.0  Asynchronous progress  Flexible message coalescing  APM support  Enhanced mpirun_rsh for scalable launching  UD-based design  Optimized and tuned collectives  Support for ConnectX  Support for QLogic/PSM  Open MPI  ConnectX support  MVAPICH2 1.0  Message Coalescing  Hot-spot avoidance  Application-initiated systems-level checkpointing  APM support  Multi-rail support for iWARP  On-demand connection management for iWARP and uDAPL  RDMA Read  Blocking

11 Mellanox Technologies OFED 1.3 Release Status  Features freeze: Oct 10, 2007  Alpha: Oct 15, 2007– kernel base:  Beta: Nov 20, 2007 – kernel base: rc2  RC1: Dec 4, 2007  RC2: Dec 18, 2007  RC3: Jan 8, 2008  RC4: Jan 22, 2008  Release: Jan 29, 2008 Note: The delay mainly caused by the move to kernel

12 Mellanox Technologies OFED 1.3 OS Matrix  kernel.org: kernel and  Novell  SLES 10  SLES 10 SP1 (up1)  Redhat  RHEL 4 (up4 and up5)  RHEL 5 (up1)  Free distros  Partial support only (no QA)  Open SuSE 10.2  Fedore C6  Ubuntu 6.06 (with RPM package installed)  Open: OSVs plans to pick up OFED 1.3, or even 1.2.5

13 Mellanox Technologies Agenda  OFED 1.3 Status  OFED Development Process  OFED Future Discussion

14 Mellanox Technologies OFED – Development Process  Start – release definition  Agree on release features  Agree on OS matrix and kernel base  Decide on target schedule  Release definition is placed on the Wiki index.php?page=OFED+Development+Wiki  Note  In all OFED releases there were changes in the above that caused release slips  Tracking meetings  Bi-weekly meetings during the development phase  Weekly meetings on the release phase  Meeting summaries are placed on the Wiki  Bug tracking in bugzilla 

15 Mellanox Technologies Kernel Development  Have one kernel git tree that is maintained by Vlad  git://git.openfabrics.org/ofed_1_3/linux-2.6.git  Cloned from Linux git tree  Add-on components (e.g., SDP) are added to the tree  New features and bug fixes are added as patches under the fixes directory  Patch convention name: _ _.patch (e.g., ipoib_0010_cq_coalescing.patch)  Supporting distros and previous kernels:  Backport directory for each kernel OFED can compile on  Add-on directory for headers from Linux  The make-dist script applies first the “fixes” patches and then the appropriate backport patches

16 Mellanox Technologies User Space Development  Each maintainer has a separate git tree on the server  Examples git://git.openfabrics.org/~sashak/management.git git://git.openfabrics.org/~shefty/libibcm.git  Each package has its own RPM SPEC file  All packages should also be released to the ofa download site   Approaching a release – a branch with the release name is opened  Userspace component fixes must be committed -- no patching scheme is used

17 Mellanox Technologies External Packages Development  External packages are provided as source RPMs  Each external package has a directory on the ofa server  Example – the bonding package: /home/monis/public_html/ofed_1_3/  latest.txt file points to the package that should be used  Ex.: ib-bonding src.rpm

18 Mellanox Technologies OFED Daily Build  An OFED package is created everyday at 6am PST   This package can be used by all to test the daily changes  This package includes  Source RPMs for kernel, all user space components, external modules  OFED scripts (install, etc.)  OFED documents  Development phase: the build uses the master branch of all git trees  Release phase: the build is based on the release branch

19 Mellanox Technologies OFED Release Phase  Starts after the alpha release  Close tracking of components status  Alpha, beta and RC are done according to the package maturity  Bugs severity tracking  which bugs must be fixed for the release  Update of all documents

20 Mellanox Technologies Agenda  OFED 1.3 Status  OFED Development Process  OFED Future Discussion

21 Mellanox Technologies OFED Goals  Enterprise Working Group within Open Fabrics Alliance (OFA)  OFED has become a brand name in the industry today  Collaborative effort to test & release OFA software  Broader test participation  Components – Kernel & User Space  Interoperability  Add-on components for vendors to differentiate above OFA Reduces deployment complexity and cost

22 Mellanox Technologies Reasons to Continue OFED  Industry synchronization point  Set of kernel modules, libraries and applications that are tested together  Comprehensive tests by many vendors  OSVs rely on our qualification work  Faster new feature development and deployment  New drivers for new IHV devices  ULP updates for new ISV software packages, e.g., Oracle 11g  Enables support of new features in existing OSes  Ex.: Bonding to work on Redhat EL4 up5, SLES 10, etc

23 Mellanox Technologies OFED-based and OS Distro-based  OFED-based and OS Distro-based distribution of OFA software – one or both  Originally, some people intended OFED to be a short-term distribution until mainstream Linux distributions caught up and were able to distribute IB/iWARP support  Today there is synergy between OFED and OS vendors as they rely on the OFED quall and support benefits delivered by commercial IB/iWARP suppliers  Are OSV’s ready to take more quall and support responsibilities at this time?  Discussion …