Fibre Channel over InfiniBand Dror Goldenberg Mellanox Technologies

2 Agenda  Background and Motivation  Protocol Overview  Software Stack Architecture

3 I/O Consolidation  Connect islands of IB to network and storage fabrics  Key factors  Gateways  Cost, performance, scalability, simplicity  Management Converge with InfiniBand OS Storage App Network App Manage- ment App OS Network App Manage- ment App IB HCA Storage App One wire GbE NIC FC HBA

4 Why FCoIB ?  Management  Similar to Fibre Channel  Transparency to applications  FC software runs over IB wires  Similar frame format / protocol  Similar driver infrastructure  Gateways - complexity reduction  SCSI transport termination  stateless packet relay  Better performance/scalability at lower cost  Better support for legacy storage attachment Gateway Host Fibre Channel Host InfiniBan d

5 FC over IB Gateway Model NPIV N_Ports FC Begins Here Physical View:  Handling initiators logins  Acts as an NPIV N_Port  Reflects F_LOGI as F_DISC Virtualized Server Logical View: NPIV HBA Fibre Channel Host FCoIB to FC Gateway Fibre Channel InfiniBand “PCIe extension”

6 FCoIB Gateway – Stateless Packet Relay  IB to FC  Strip IB headers  Payload contains entire FC frame  FC to IB  D_ID lookup to retrieve addressing {LID, QPN, etc.}  Encapsulate FC frame as UD payload IB CRCIB Headers FC Frame IB CRCIB Headers FC Frame

7 FCoIB Protocol  Encapsulation of Fibre Channel frames over IB transport  FC-0 Physical is replaced by IB Physical  FC-1 Encode/Decode, Framing and Link Control are replaced by IB link and framing  Degenerated delimiters are encapsulated  Similar to FCIP/iFCP RFC 3821/4172  Routing according to IB addresses (LID/GID/QP)  FC-2 and above remain intact  Similar concept to IP over IB encapsulation (RFC 4391)  FCoIB HBA is an IB UD application

8 Frame Encapsulation Encapsulated FC Frame VCRC DETH LRH DestQP field indicates FC FC Payload (0 to 2112 bytes) FC Header (24 bytes) Enc. SOF (4 bytes) FC CRC (4 bytes) Enc. EOF (4 bytes)  Destination QP indicates FC Nx_Port  FCoIB HBA is an IB UD application  Gateway maintains mapping between FCID and IB address ICRC BTH IB Headers IB CRCs

9 Discovery  Identifying the Gateway  Through an IB specific mechanism (TBD)  e.g. Service Record, IO Controller, Multicast  Results in {GID,QPN} of the gateway  FLOGI between IB initiator and the gateway  Gateway initiates an F_DISC with the fabric  Other frames can be relayed intact

10 Software Stack Architecture Open FC mlx4_en TCP/IP SCSI midlayer mlx4_core mlx4_ib IB Core mlx4_fc IB Client (Discovery) send() - non offloaded scsi_send() - offloaded I/O

11 mlx4_fc HW Interface  FC Raw Frame Interface (RFCI)  Sends and receives FCoIB frames  FC CRC offload  FC Command Queues (FCMND)  Initiate IO operation  Owns bulk of FEXCH Queues  FC Exchange Queues (FEXCH)  Data transfer  Statically allocated to FCMND queues  Pre allocated chunk of queues

12 mlx4_fc HW Interface  FC Memory Keys  Single/dual MKey per FEXCH  Pre-allocated chunk of Data MKeys  Used to register buffers for IOP  Zero Copy Operations  Completions, Events, Interrupts, Command interface  As usual

13 mlx4_core  Resource broker  Allocation of chunks of QPs/MRs to upper drivers  Command interface serialization

14 Example IO Read Flow FCP_CMND FCP_DATA FCP_RSP IORead CMD Allocate xChg Map DMA Area Register memory Issue CMD openfcmlx4_fc Zero Copy HW DMA Response IB wireConnectX IO Read Done Release xChg Unmap DMA

15 Summary  FCoIB – FC frames Encapsulation over IB UD  FCoIB is a UD application  Simplifies gateway implementation  Performance, cost, scalability  Simplifies software and management  FC look & feel  Protocol is fully offloaded in ConnectX