1. Network speed tests
CERN 14-dec-2010

2. Hardware used 2 identical servers:
Dell PowerEdge T610 with 2x Intel xeon X5650, 24GB ram
2x PCI-Ex x8, 2x PCI-Ex x4
2x10Gb on copper cards, Broadcom chip (NetXtreme II ), PCI-Ex x8
1 switch Dell PowerConnect 6224:
24 Gb ports plus 2x10Gbase-T ports module

3. IPerf test tcp benchmark, bi-directional
do not really expect to use tcp, but still a good hw test
very similar results using one-way or bi-directional transfers
identical results with or without passing through the switch

4. IPerf results ~9.2 - 9.3 Gb/s with switch ~9.3 - 9.4 Gb/s without
[5] sec 1.09 GB 9.37 Gbits/sec [5] sec 10.8 GB 9.31 Gbits/sec (one direction)
[4] sec 1.09 GB Gbits/sec [4] sec 10.7 GB 9.21 Gbits/sec (other direction)
~ Gb/s
with switch
~ Gb/s without

5. UDP results Essentially same (disappointing) performance with
HPCBENCH tool
# UDP communication test -- Mon Dec 13 23:14:
# Fixed packet size unidirectional stream test
# Hosts: gtktest1 (client) <--> (server)
# Client UDP socket buffer size (Bytes) -- SNDBUF: RCVBUF: 65508
# Server UDP socket buffer size (Bytes) -- SNDBUF: RCVBUF: 65508
# Client IP TOS type: Default
# Server IP TOS type: Default
# UDP datagram (packet) size (Bytes) -- Client: 1460 Server: 1460
# Data size of each read/write (Bytes) -- Client: 1460 Server: 1460
# Message size (Bytes): 16384
# Test time (Second):
# Test repeat: 1
# Network(Mbps) Local(Mbps) SentPkg(C) SentByte(C) RecvPkg(S) RecvByte(S) LostPkg LossRate
Essentially same (disappointing) performance with
HPCBENCH tool
custom (very basic) client-server python application

6. UDP vs. TCP on PCI-Ex 4x slot
Due to wrong mounting (by Dell…), tests using the 10 GB cards in the PCI-Ex 4x slots (in place of 8x)
Degraded performance due to reduced PCI-Ex bandwidth
TCP
UDP
Mbit/s
Bandwidth vs. Message size
Bandwidth (average/min./max.) vs. Message size
Bytes

7. More investigations needed!
UDP strange results
UDP performance is the same, Gbits/sec (slightly depending on packet size), in both configurations (“wrong”: cards in the PCI-Ex 4x slots, and “correct”: cards in the PCI-Ex 8x slots)
We’re hitting some kind of wall:
Collisions?
Lost packets?
Optimization of frame-buffer?
More investigations needed!

8. UDT (UDX) Just started investigating !!
UDP based application level data transfer protocol
adds own reliability control and congestion control
highly configurable, still looking at the possibilities
not claiming we will use this protocol, but if we want reliability control udp is a candidate

9. Still to do Lots of work; just to mention:
understand if we need reliability control
test behaviour with multiple 1Gb senders/one 10Gb receiver
avoid memory-to-memory copy
etc etc