1. 1 LOFAR Network Experiences 20-9-2007 LOFAR – Wide Area Network LOFAR – Network Experiences Klaas Stuurwold Roel Gloudemans Peter Maat

2. 2 LOFAR Network Experiences 20-9-2007 LOFAR Network Overview LOFAR – Arms ~ 45 Stations 2.5Gb/s per station in 4 approx. 600 Mb/s data streams Max distance Station to core is 80km LOFAR – Arms ~ 45 Stations 2.5Gb/s per station in 4 approx. 600 Mb/s data streams Max distance Station to core is 80km LOFAR – Core ~ 32 Stations 2 - 12Gb/s per station in max 12 1Gb/s Mb/s data streams Max distance Station to core is 80km LOFAR – Core ~ 32 Stations 2 - 12Gb/s per station in max 12 1Gb/s Mb/s data streams Max distance Station to core is 80km Monitoring and control ~ 100Mb/s per station LOFAR Partner data ~ 100Mb/s per station Max latency: 10ms Availability min. 95% Upgradeable: Longer arms Higher data rates Monitoring and control ~ 100Mb/s per station LOFAR Partner data ~ 100Mb/s per station Max latency: 10ms Availability min. 95% Upgradeable: Longer arms Higher data rates Data characteristics Raw Ethernet or IP/UDP Unidirectional Jumbo frames Per station a few large data streams Point to point connections Data characteristics Raw Ethernet or IP/UDP Unidirectional Jumbo frames Per station a few large data streams Point to point connections

3. 3 LOFAR Network Experiences 20-9-2007 Initial approach: I Use COTS equipment Experience readily available Cheap Short order to deployment time Flexible Use 1Gb/s technology combined with CWDM Cheapest option Use switches and no media converters Optimal flexibility, no replugging needed on RFP board or port Cheap, one switch with multiple ports costs about as much as one media converter However, networks for different goals not isolated by default Use VLAN’s to separate data from management and partners Separate IP address spaces possible Separation of network traffic Bandwith control or Quality of Service per VLAN possible on most switches Use multi-link trunking to combine several 1Gb/s ports into one logical port No need for Spanning tree protocol which generates network overhead One single logical interface to configure

4. 4 LOFAR Network Experiences 20-9-2007 Initial Approach: II See Poster

5. 5 LOFAR Network Experiences 20-9-2007 Unexpected Challenges: I Configuration errors might make a remote station unaccessible. In the beginning it was thought that not much configuration would be needed In practice, changes where happening every week if not day Result: Regular occurrence of “terminal” miss-configuration. Solution: Add “out-of-band” management equipment for switches and servers For network equipment a separate data-path is needed RS-232 to Ethernet boxes are readily available. Often with extended features which prove to be extremely useful. e.g. the possibility to connect door switches and temperature sensors The Jumbo frame “standard” Jumbo frames can be defined according to the Ethernet II standard, or the IEEE802.3 standard The difference: In the IEEE standard the VLAN tag is included in the 9000 bytes, thus the data frame must be smaller. Result: No data transmission if part of the network is using the Ethernet II standard with max data frame size. Solution: Check specs carefully before buying.

6. 6 LOFAR Network Experiences 20-9-2007 Unexpected Challenges: II Real Unidirectional traffic The data receiving stations do no transmit any ethernet traffic. This prevents the switches from learning their MAC addresses. Thus all data is broadcasted on all ports So when one station starts transmitting the station becomes unreachable (if we didn’t have VLAN’s the whole network would become unreachable) Solutions: Create a static MAC address table on the switch. Surprisingly few switch brands support this. It is also error prone. Schedule a network ping on the data stations. Works very well, but is not possible with a Blue Gene The tree is not fair We want our switches to be able to operate on full speed on all ports Nowadays this is possible on most 1Gb/s switches if you plug right (Cisco is the most famous exception) On the switch mainboard there is often one ASIC per couple of ports. These ports can communicate to each other on full speed all the time. However these ASIC’s are grouped under another ASIC, this puts a limitation on ASIC to ASIC traffic. Some of these switches are sold as Full Speed Non-Blocking! Solution: Read switch specs very careful and agree on a return policy. Don’t trust cheap switches!

7. 7 LOFAR Network Experiences 20-9-2007 Unexpected Challenges: III Multi-Link trunking, is the pain worth the gain? One sender-receiver data stream cannot be balanced over the links in a trunk Thus, on a trunk consisting of 2 links, it is only possible to have 2 600Mb/s data streams and not 3. Data streams are not balanced over the links according to the current load of the link. An algorithm with sender-receiver MAC/IP addresses or protocol numbers is used. Solutions: Use 10Gb/s networking Choose source and destination addresses wisely. Obtain the balancing algorithm from the vendor. (Very often this is classified information!) Buy cheap, pay later In our experience the firmware of cheap switches often contain blocking bugs The performance of cheap switches is not always what you’d expect from the documentation. Fair tree and/or CPU issues The better (more expensive) switches often have features that help you overcome unexpected challenges Price and MTBF are related

8. 8 LOFAR Network Experiences 20-9-2007 The 10G story: I The results are preliminary. We had some problems with our special “Glow-in-the-dark” ™ fibre. This limited the connection speeds to max 8Gb/s. 10G networking, a solution for: Trunking; not needed at station level for LOFAR. One 10G lightpath is an ideal mechanism to connect stations abroad. The amount of fibres/muxes needed per station Built in overhead for future growth Less maintenance/less complicated infrastructure At the same price (or less!). Short range (10km) optical at 2000 euro per switch port Modern premium brand switches are able do 10Gb/s full speed non blocking. The addition of 2 10Gb/s switches between 2 computer systems which where connected directly before only added 0.3ms extra delay Modern PCI-X based systems are able to almost saturate a 10Gb/s link with TCP/UDP traffic if protocol offloading cards are used. (e.g. Myrinet cards; use Jumbo frames!).

9. 9 LOFAR Network Experiences 20-9-2007 The 10G story: II Connections between networks often require filtering or some form of address translation We were not able to to find any differences in network performance when filter rules where added to the switch. When filtering was done at the PC level, with <10 rules no difference, but with 255 rules, the performance dropped to 7.8Gb/s Fibre quality is important. A fibre which performs well on 1Gb/s might not perform with 10Gb/s. No clear problems could be detected except for a few bad packets and inconsistent test results.