1. Storage area network and System area network (SAN)
What are they?
Network requirements
Hardware/software issues
References:
Ulf Troppens, Rainer Erkens, a nd Wolfgang Muller, “Storage Networks Explained - basic and application of Fibre Channel SAN, NAS, iSCSI and Infiniband”, John Wiley & Sons,
W. J. Dally and B. Towles, “Principles and Practices of Interconnection Networks”, Morgan Kaufmann, 2004.
Ajay V. Bhatt, “Creating a Third Generation I/O Interconnect,” available at

2. Storage area network (SAN):
Server-centric IT architecture: storage devices exist only with servers

3. Storage-centric IT architecture: SCSI cables are replaced by a network (storage is now independent of servers).

4. Storage area network (SAN) requirement:
Serial transmission for high speed and long distance
Low transmission errors
Low delay of transmitted data
Needs to make it feel like using a local disk
Low delay is a relative term:
The disk subsystem has around 1ms – 10ms latency itself.
The communication protocol should not use CPU.

5. Current Storage area network (SAN) technology (IBM):
Fibre Channel
TCP/IP + Gigabit Ethernet (iSCSI)
InfiniBand

6. System area network: a network with a high bandwidth and a low lantency that serves as a connection between computers in a distributed computer system.

7. Why system area network:
Historically, the system area network comes with a particular parallel machine (supercomputer, e.g. Cray T3D, Cray T3E, SGI origin 2000, IBM SP, Thinking machine CM5, Intel Polygon)
The network is very expensive due to low volume
CPU is two generations behind
A more cost effective way to build these system is to decouple the processor technology from the networking technology.
To form cheaper clusters of workstations with the off-the-shelf system area network technology (compared to traditional supercomputers).
current system area networks:
Myrinet, Quadrics, Infiniband

8. System area network requirement:
Low latency and high bandwidth at the application level.
Not just at the hardware level
Not just at the system level
Implicitation:
Hardware, network interface, software messaging layer should work together to achieve the goal.
Infiniband is designed as both storage area network and system area network.

9. Hardware issues: High speed links:
Infiniband: 2.5Gbps = 250MBps, 10Gbps=1GBps, 30 Gbps = 1GBps
Fibre channel: 100MBps, 200MBps, 400MBps, 1GBps.
Myrinet: up to 9.6Gbps
As a reference PCI bus: 100MBps
NIC may need to attach to the memory bridge

10. A typical PC:

11. A workstation connected to a system area network:

12. When the number of end points is large, multiple switches will be needed.
Topology
Switching
Routing

13. Topology
Static arrangement of channels and nodes in an interconnection network
Trade-off between cost and performance
Cost: the number and complexity of chips, density and length of the interconnections, etc.
Performance:
Bandwidth and latency: also depend on other factors other than topology
Topology performance metrics: Bisection bandwidth, diameter, nodal degree, channel load

14. A cut of a network is the set of channels that partitions the set of all nodes into two disjoint sets.
A bisection of a network is a cut that partitions the network nodes in roughly half.
The bisection bandwidth of a network is the minimum bandwidth over all bisections of the network.
The diameter of a network is the largest minimal hop count over all pairs of nodes.
Under a particular traffic pattern, the channel that carries the largest fraction of traffic determines the maximum channel load of the topology.

15. Example topologies: Regular or irregular
Regular topologies are mostly derived from two main families: butterflies (k-ary n-flies) or tori (k-ary n-cubes)

16. Switching: how a packet pass a switch
Message/packet/flit

17. Traditional scheme: store-and-forward
Time = H (S + P/B)

18. Cut-through switch:
Forward to the next link after the header flit is received. Stop only when the next hop buffer is not available.
Time = H S + P/B, when S << P/B, the time does not depend on the number of hops!!!

19. Wormhole routing:
Cut-through switches still allocate buffer to packets. May require a large amount of buffers
Wormhole routing only allocates buffer for one flit for each packet.
Latency is the same as cut-through switching.
When the packet is block, the whole flit “train” is block, occupying links.
Solution: add more virtual channels.

20. The deadlock problem in wormhole routing:
Need deadlock free routing scheme to select the right path

21. Cut-through switch and wormhole switch are widely used in system are networks
Routing in such systems is an issue!!
Shortest path routing may result in deadlock.
Deadlock free routing:

22. Cut-through switch and wormhole switch are widely used in system are networks
Routing in such systems is an issue!!
Shortest path routing may result in deadlock.
Deadlock free routing:
Basic idea: fix the priority of channels and using the channels with increasing priority.
Example: up/down routing

23. Up/down routing: Select a node as the root
Build a spanning tree from the root
Nodes are partitioned into layers based on the position in the spanning tree
The channel from a lower layer node to a higher layer node is the up link, the channel from a higher layer node to a lower layer node is a down link, channels between nodes in the same layer are marked as up or down link based on the node number
In the valid route: an up channel cannot follow an down channel.
These exists at least one valid path between each pair of nodes.

24. Problems with deadlock free routing:
Load balancing is a problem, traffic are not evenly distributed
Non-adaptive version of the deadlock free routing scheme is also a problem
How to map the routes in order to get good performance (metrics: maximum channel load?)
More on the problem to be discussed later.

25. Hardware/software codesign and software API issues:
What functionality should be implemented in the hardware.
E.g. adaptive routing may imply out of order packets
Chien’04 paper gives good answers to some of these questions.