Infiniband subnet management Discuss the Infiniband subnet management system Discuss fat tree and subnet management in an Infiniband with a fat tree topology. References A. Bermudez, R. Casado, F.J. Quiles, T. M. Pinkston, J. Duato, “Evaluation of a Subnet Management Mechanism for Infiniband Networks”, ICPP A. Vishnu, A. R. Mamidala, H. Jin, D. K. Panda, “Performance Modeling of Subnet Management on Fat Tree Infiniband Networks using OpenSM”, Workshop on System Management Tools on Large Scale Parallel Systems, Held in Conjunction with IPDPS 2005 X. Lin, Y. Chung, T. Huang, “A Multiple LID Routing Scheme for Fat-Tree- Based Infiniband Networks.” IPDPS 2004.

Infiniband devices and entities related to subnet management Devices: Channel Adapters (CA), Host Channel Adapters, switches, routers Subnet manager (SM): discovering, configuring, activating and managing the subnet A subnet management agent (SMA) in every device generates, responses to control packets (subnet management packets (SMPs)), and configure local components for subnet management SM exchange control packets with SMA with subnet management interface (SMI).

Subnet management packets (SMP) –256 bytes of data –Use unreliable datagram service on the management virtual lane (VL 15) –LID routed: use lookup table for forwarding Use after the subnet is setup. E.g. Check the status of an active port –Direct routed: has the information of the output port for each intermediate hop. Subnet discovery for the subnet is setup

Subnet management packets (SMP) –Define the operation to be performed by SM –Get: get the information about CA, switch, port –Set: set the attribute of a port (e.g. LID) –GetResp: get response –Trap: inform SM about the state of a local node A SMA stop sending Trap message until it receives TrapRepress packet. Topology information can be obtained by a sweep and by peridical Traps.

Subnet Management phases: –Topology discovery: sending direct routed SMP to evert port and processing the responses. –Path computation: computing valid paths between each pair of end node –Path distribution phase: configuring the forwarding table

Subnet discovery –SM starts by sending a direct routed Get SMP to its local node. Upone receiving response, SM sends SMPs with additive depth.

Path computation: –Compute paths between all pair of nodes –For irregular topology: Up/Down routing does not work directly –Need information about the incoming interface and the destination and Infiniband only uses destination –Potential solution: »find all possible paths »remove all possible down link following up links in each node »find one output port for each destination –Why is that still working? No clear to me. –Other solutions: destination renaming –Fat tree topology: What is the best that can be achieved is also not clear.

Path distribution: –Ordering issue: the network may be in an inconsistent state when partially updated, which may result in deadlock during this period. Traditional solution, no data packets for a period of time deadlock free reconfiguration schemes.

Fat Tree: –A way to build large scale clusters

Fat Tree: –Routing in a complete fat tree: a up phase and a down phase (always contention free).

Fat Tree: –The complete fat tree has the scalability problem The root has a very large nodal degree –How to build a fat tree with nodes that have a constant nodal degree. –M-port n-tree FT(m, n) m is the number of port per switch n+1 is the height of the tree The tree consists of 2*(m/2)^n processing nodes and (2n-1)*(m/2)^(n-1) switches.

–How is an m-port n-tree FT(m, n) connected? m is the number of port per switch n+1 is the height of the tree The tree consists of 2*(m/2)^n processing nodes and (2n-1)*(m/2)^(n-1) switches. A processing node is labeled as P(p_0 … p_{n-1}), –P_0 = 0..m-1, p_i (i!=0) = 0..m/2-1 A switch is labeled as SW –l = 0..n-1, –When l=0, w_i = 0..m/2-1 –When l!=0, w_0 = 0..m-1, w_I (I!=0) = 0..m/2-1

Figure 5: a 4-port 3-tree

How is the tree connected? –SW _k be the kth port of SW(w, l). –SW(w, l)_k and SW(w’, l’)_k’ is connected iff l’ = l + 1 w_0…w_{n-3} = w_0’…w_{l-1}’w_{l+1}’…w_{n-2}’ k=w_l’, k’ = w_{n-2} + m/2 –Question: which switches are connected to SW ? How the ports are connected?

Fat tree properties: –Multiple routes between two nodes Deterministic routing: one path between two nodes, how to map? –What is a good mapping? In the case when the traffic pattern is unknown, common practice is to minimize the maximum load on a link. –Do we know how to do it? Not clear even when there is no restriction on the routing. It is likely that an optimal solution exists for a particular FAT tree topology In infiniband, destination based routing put some restriction on which path can be used.