Enhancements for Voltaire’s InfiniBand simulator By: Alex Netes Supervisors: Prof. Ran Giladi, Mr. Alex Seltser

Outline InfiniBand overview Voltaire overview InfiniBand simulator overview Project objectives Porting simulator on host Overview Implementation notes Status Configuration file generator Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

InfiniBand Overview InfiniBand, a high performance serial switched system interconnect industry standard, was released in July 2000. Its transport protocol, which was originally designed for implementation in wire speed silicon state machines, removes any unnecessary features that might harm the performance or latency, enabling support of bandwidths ranging from 2.5Gb/S up to 30Gb/S. InfiniBand architecture defines the entire stack from the physical to the application layers API’s and fabric management. InfiniBand has unique capabilities for direct data placement implemented by RDMA and for OS bypassing, discovery, fail-over, remote boot, I/O sharing etc’. Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

InfiniBand Overview The InfiniBand system elements : HCA (Host Channel Adapter), an adapter or silicon chip residing on the host motherboard providing the host side functionality and advanced reliable message passing capabilities. Switch, connecting all the nodes in a switched point to point manner. Router, providing connectivity to other InfiniBand or IP subnets. Subnet Manager (SM), subnet management entity, responsible for fabric configuration – local addresses assignment, maintaining routing tables, QoS, etc’. Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

InfiniBand Overview InfiniBand subnet network HCA SM HCA Switch HCA Router HCA Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

InfiniBand Overview Subnet Manager state machine: Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

InfiniBand Overview Addressing: LIDs, Local IDentifiers, are subnet-unique 16-bit identifiers used within a network by switches for routing. GUIDs, Global Unique IDs, are 64-bit EUI-64 IEEE defined identifiers for elements in a subnet. GIDs, Global IDs, are 128-bit global identifiers used for routing across subnets. Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

InfiniBand Overview Links properties: State: Active, Down Width: 1x, 4x, 12x Speed: 2.5Gb/s, 5.0Gb/s Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Voltaire overview Multi-Protocol Switches (InfiniBand, FC and IP Ports) ISR 6000 FC and IP Ports Modules ISR 9024 ISR 9096 ISR 9288 Management Server Products InfiniBand PCI Express & PCI-X Adapters InfiniBand Software Stack VoltaireVision Management Suite Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Voltaire overview Voltaire deployed 6 out of 10 InfiniBand clusters in Top500 NASA:10,240 CPUs Most powerful Production Supercomputer in the world Dozens of grids and clusters with over 100 CPUs Combining compute and scalable File I/O Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

InfiniBand simulator overview One of the tools, developed for internal use is the InfiniBand fabric simulator. The simulator is used by several departments and in different stages of software development and marketing. The main task of the simulator is to create virtual InfiniBand fabric, consisting of large number of Voltaire's switches and HCAs for developing and debugging purposes. Many designing and development aspects could be tested only using the simulator such as routing algorithm behavior, nodes failures, fabric configuration, etc'. Moreover, simulator is a key utility in debugging problems in real functioning fabrics, used by the R&D team. Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

InfiniBand simulator overview Simulator structure: Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Project goals QA and R&D requirements. Improve fabric creation process. Add support for additional modules. Add support for multi-SM fabric simulation. Port simulator on host. Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Porting simulator on host Simulator module: Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Porting simulator on host Implementation notes: Added INET sockets support. IBSim server IBSim client Ctl_socket ctlfd Netcon[i] fd UDP 8000 UDP X Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Status Solution is working on a Open source version of the simulator. The solution for Voltaire’s version of the simulator is in the last stages of development. Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Configuration file generator Define language, that enables to define any InfiniBand object and fabric. Should be simple on one hand and very flexible on the other. Create parser for this language, that will transform it to simulator files format. Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Configuration file generator Implementation notes: Theoretical information from CS compilation course. YACC & LEX (Bison & Flex) based parser. Reserved words: create, extends, param, link, map. Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Configuration file generator - example File: anafa.cfg create anafa( param (port_num=24) param (node_type=switch) ) Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Configuration file generator - example File: slb24.cfg #Include “anafa.cfg” create slb24( extends anafa[1-2]( param (devid=0x56a8) ) param (port_num=24) map (1=anafa[1].13) … Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Configuration file generator - example File: sfb4.cfg #Include “anafa.cfg” create sfb4( extends anafa[1]( param(devid=0x31a9) ) param (port_num=24) Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Configuration file generator - example File: isr9096.cfg #include “slb24.cfg” #include “sfb4.cfg” create isr9096( extends slb24[1-4]() extends sfb4[1-4]() param (port_num=96) link (slb24[1].anafa[1].1=sfb4[1].anafa[1].1) … map (1=slb24[1].1) ) Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Configuration file generator - example File: hca.cfg create hca( param (port_num=2) param (node_type=hca) ) Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Configuration file generator - example File: fabric.cfg #include “isr9096.cfg” #include “hca.cfg” create fabric( extends isr9096[1]( extends slb24[2,4]() extends sfb4[1]() ) extends hca[1-48]() link (isr9096.1[1-48]=hca[1-48]) Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Status Parser is in last stages of debug. Enhancements for Voltaire's InfiniBand simulator. 2007. Communication System Engineering.

Thank you.