Infiniband Architecture Aniruddha Bohra 1/29/2002 CS 545 - Distributed Systems

Distributed Applications and Data Transfer Traditional distributed applications Need low latency message delivery Data volume in transfers between nodes not too high Server applications Need low latency and high bandwidth data transfers Data volumes in transfers are high e.g. in a cluster based storage or streaming multimedia servers Need Reliable and Available Services Need easy maintenance 1/29/2002 CS 545 - Distributed Systems

Traditional message send To NIC Application Memory buffers System Call Kernel TCP sendmsg Copy from user space IP and lower layers Backup buffers One kernel boundary crossing Two memory copies!! 1/29/2002 CS 545 - Distributed Systems

Lessons from parallel computing Co-processors that can access memory directly used for communication FLASH, J-Machine, Alewife User level networking Virtual Memory Mapped Communication Unet VMMC VIA 1/29/2002 CS 545 - Distributed Systems

Interconnect bottleneck Servers require high data transfer rate CPUs operate at GHz speed Gigabit ethernet is commonly used in cluster based servers Data volumes are high PCI bus is much slower operates at 32 bit/33 MHz or 64 bit/66 MHz the next generation bus PCI-X operates at 133 MHz 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Some solutions HyperTransport Runs at 800MHz full duplex Bridges with current buses and other HyperTransport buses 3GIO Switch based Provides a layered implementation Promises more than 40 Gb/s transfer rate 1/29/2002 CS 545 - Distributed Systems

More problems with bus based interconnects Cannot keep up with the increasing CPU and peripheral speed Bus is shared between all peripherals The pin count is high – PCB space is limited! Buses are not able to extend to long distances Do not support a large number of devices 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Outline Motivation and background Infiniband architecture Infiniband components Infiniband operation Other Infiniband features Status Summary 1/29/2002 CS 545 - Distributed Systems

Infiniband Architecture Provides switch based interconnect Increased reliability Scalable and easily maintainable Supports memory to memory communication Low latency communication Provides support for “out of box” components Scalable Easier to manage and operate Is complimentary to the 3GIO and HyperTransport Buses 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems What is Infiniband? Infiniband Architecture(IBA) defines a System Area Network (SAN) IBA SAN is a communications and management infrastructure for I/O and IPC IBA defines a switched communications fabric high bandwidth and low latency protected, remotely managed environment. IBA hardware off-loads from the CPU much of the I/O communications operation. 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems An IBA SAN 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Outline Motivation and background Infiniband architecture Infiniband components Infiniband operation Other Infiniband features Status Summary 1/29/2002 CS 545 - Distributed Systems

Topologies and components IBA serves as an interconnect for endnodes A node can be a processor node, an I/O unit and/or a router to another network Node Node Infiniband Fabric Node Node Node Node Node 1/29/2002 CS 545 - Distributed Systems

Topologies and Components An IBA network is subdivided into subnets interconnected by routers Endnodes can attach to a single or multiple subnets An IBA subnet is composed of endnodes, switches, routers and subnet managers Each IBT device may attach to a single switch or multiple switches and/or directly with each other 1/29/2002 CS 545 - Distributed Systems

IBT device – processor node Verbs Consumer Consumer Consumer Message and Data Service Channel Adapter (endnode) Port Channel Adapter (endnode) Port 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Processor node Each channel adapter constitutes a node on the fabric Architecture supports multiple channel adapters per unit with each adapter providing one or more ports to the fabric Message and Data service is an OS component Verbs describe the functions to configure, manage and operate a host channel adapter Verbs are not API but provide the framework for OS to specify it 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Channel Adapter An IBA channel adapter(CA) is a programmable DMA engine with special protection features that allow DMA operations to be initiated locally and remotely. Host Channel Adapter(HCA) provides a consumer interface providing the functions specified by IBA verbs. Target Channel Adapter(TCA) provides an interface to the device 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Channel Adapter 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Addressing in IBA Each endnode has one or more CAs and each CA has one or more ports Each Queue Pair (QP) has a QP number (QPN) assigned by the CA Each port has a unique Local ID (LID) and at least one IPv6 address – Global ID (GID) 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Switches Do not generate or consume packets – pass them along based on the destination address Are the routing components for intra-subnet routing – support uni or multicast Every destination is configured with one or more unique Local IDs (LIDs) Subnet manager configures switches including loading their forwarding tables 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Routers Routers are inter-subnet routing elements Routers forward packets based on the packet’s global route header Routers expose one or more ports between which packets are relayed IPv6 specifies the protocol performed between routers to derive their routing tables 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Subnet Managers An Subnet Manager(SM) is an entity attached to a subnet responsible for its management Tasks Discover topology Configure the CA port with a range of LIDs, GIDs, subnet prefix and Partition_Keys Maintains LID/GID resolution tables 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Outline Motivation and background Infiniband architecture Infiniband components Infiniband operation Other Infiniband features Status Summary 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Communication Queuing Consumer queues up a set of instructions for hardware to execute (Work queue). Work queues are created in pairs(Queue pairs – QP) for send and receive operations Each Work Queue has corresponding Completion Queue 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Work Queue Operations Send operations SEND Block in memory space to send to destination RDMA RDMA_READ, RDMA_WRITE, ATOMIC Memory Binding Alters the memory binding relationship – gives the R_KEY to components which allows secure DMA Receive operation Specifies a receive data buffer 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Work Queue Operations 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Communication Stack 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Keys Keys are used to provide isolation and protection M_KEY Enforces the control of a master Subnet Manager B_KEY Enforces control of a baseboard Subnet Manager P_KEY Enforces membership in a subnet Q_KEY Enforces access rights for reliable or unreliable service L_KEY and R_KEY Provide access rights to Remote registered memory 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Outline Motivation and background Infiniband architecture Infiniband components Infiniband operation Other Infiniband features Status Summary 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Virtual Lanes A virtual lane represents a set of transmit and receive buffers in a port VL15 is used for subnet management Each port must have at least one data VL Separate flow control is maintained over each VL 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Service Levels Service levels(SLs) are maintained by attaching a VL to a SL IBA does not specify any QoS levels(e.g. best effort) The SMA must keep a mapping of Service Level to Virtual Lane and propagate it through the switch 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Status Intel Developer Forum had several status talks http://www.intel.com/idf/us IBA enabled network storage has been demonstrated at industry shows Banderacom Windriver The first products are expected to be in the market by middle of 2002 1/29/2002 CS 545 - Distributed Systems

CS 545 - Distributed Systems Summary Future bandwidth requirements for servers would lead to the interconnect becoming a bottleneck – IBA is an attempt to alleviate the problem IBA provides a thorough migration from a bus based to a switch based architecture while maintaining interoperability Further deployment is needed to realize other issues that would arise in operation 1/29/2002 CS 545 - Distributed Systems