1. Enhancements for Voltaire’s InfiniBand simulator
By: Alex Netes
Supervisors: Prof. Ran Giladi, Mr. Alex Seltser

2. Outline InfiniBand overview Voltaire overview
InfiniBand simulator overview
Project objectives
Porting simulator on host
Overview
Implementation notes
Status
Configuration file generator
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

3. InfiniBand Overview
InfiniBand, a high performance serial switched system interconnect industry standard, was released in July Its transport protocol, which was originally designed for implementation in wire speed silicon state machines, removes any unnecessary features that might harm the performance or latency, enabling support of bandwidths ranging from 2.5Gb/S up to 30Gb/S.
InfiniBand architecture defines the entire stack from the physical to the application layers API’s and fabric management. InfiniBand has unique capabilities for direct data placement implemented by RDMA and for OS bypassing, discovery, fail-over, remote boot, I/O sharing etc’.
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

4. InfiniBand Overview
The InfiniBand system elements :
HCA (Host Channel Adapter), an adapter or silicon chip residing on the host motherboard providing the host side functionality and advanced reliable message passing capabilities.
Switch, connecting all the nodes in a switched point to point manner.
Router, providing connectivity to other InfiniBand or IP subnets.
Subnet Manager (SM), subnet management entity, responsible for fabric configuration – local addresses assignment, maintaining routing tables, QoS, etc’.
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

5. InfiniBand Overview InfiniBand subnet network HCA SM HCA Switch HCA
Router
HCA
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

6. InfiniBand Overview Subnet Manager state machine:
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

7. InfiniBand Overview Addressing:
LIDs, Local IDentifiers, are subnet-unique 16-bit identifiers used within a network by switches for routing.
GUIDs, Global Unique IDs, are 64-bit EUI-64 IEEE defined identifiers for elements in a subnet.
GIDs, Global IDs, are 128-bit global identifiers used for routing across subnets.
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

8. InfiniBand Overview Links properties: State: Active, Down
Width: 1x, 4x, 12x
Speed: 2.5Gb/s, 5.0Gb/s
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

9. Voltaire overview
Multi-Protocol Switches (InfiniBand, FC and IP Ports)
ISR 6000
FC and IP Ports Modules
ISR 9024
ISR 9096
ISR 9288
Management
Server Products
InfiniBand PCI Express & PCI-X Adapters
InfiniBand Software Stack
VoltaireVision
Management Suite
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

10. Voltaire overview
Voltaire deployed 6 out of 10 InfiniBand clusters in Top500
NASA:10,240 CPUs
Most powerful Production Supercomputer in the world
Dozens of grids and clusters with over 100 CPUs
Combining compute and scalable File I/O
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

11. InfiniBand simulator overview
One of the tools, developed for internal use is the InfiniBand fabric simulator. The simulator is used by several departments and in different stages of software development and marketing. The main task of the simulator is to create virtual InfiniBand fabric, consisting of large number of Voltaire's switches and HCAs for developing and debugging purposes. Many designing and development aspects could be tested only using the simulator such as routing algorithm behavior, nodes failures, fabric configuration, etc'. Moreover, simulator is a key utility in debugging problems in real functioning fabrics, used by the R&D team.
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

12. InfiniBand simulator overview
Simulator structure:
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

13. Project goals QA and R&D requirements.
Improve fabric creation process.
Add support for additional modules.
Add support for multi-SM fabric simulation.
Port simulator on host.
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

14. Porting simulator on host
Simulator module:
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

15. Porting simulator on host
Implementation notes:
Added INET sockets support.
IBSim server IBSim client
Ctl_socket ctlfd
Netcon[i] fd
UDP 8000
UDP X
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

16. Status Solution is working on a Open source version of the simulator.
The solution for Voltaire’s version of the simulator is in the last stages of development.
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

17. Configuration file generator
Define language, that enables to define any InfiniBand object and fabric.
Should be simple on one hand and very flexible on the other.
Create parser for this language, that will transform it to simulator files format.
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

18. Configuration file generator
Implementation notes:
Theoretical information from CS compilation course.
YACC & LEX (Bison & Flex) based parser.
Reserved words: create, extends, param, link, map.
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

19. Configuration file generator - example
File: anafa.cfg
create anafa(
param (port_num=24)
param (node_type=switch) )
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

20. Configuration file generator - example
File: slb24.cfg
#Include “anafa.cfg”
create slb24(
extends anafa[1-2](
param (devid=0x56a8) )
param (port_num=24)
map (1=anafa[1].13) …
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

21. Configuration file generator - example
File: sfb4.cfg
#Include “anafa.cfg”
create sfb4(
extends anafa[1](
param(devid=0x31a9) )
param (port_num=24)
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

22. Configuration file generator - example
File: isr9096.cfg
#include “slb24.cfg”
#include “sfb4.cfg”
create isr9096(
extends slb24[1-4]()
extends sfb4[1-4]()
param (port_num=96)
link (slb24[1].anafa[1].1=sfb4[1].anafa[1].1) …
map (1=slb24[1].1) )
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

23. Configuration file generator - example
File: hca.cfg
create hca(
param (port_num=2)
param (node_type=hca) )
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

24. Configuration file generator - example
File: fabric.cfg
#include “isr9096.cfg”
#include “hca.cfg”
create fabric(
extends isr9096[1](
extends slb24[2,4]()
extends sfb4[1]() )
extends hca[1-48]()
link (isr9096.1[1-48]=hca[1-48])
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

25. Status Parser is in last stages of debug.
Enhancements for Voltaire's InfiniBand simulator Communication System Engineering.

26. Thank you.