1. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 1 10 Gigabit Ethernet Test Lab PCI-X Motherboards Related work & Initial tests Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” www.hep.man.ac.uk/~rich/

2. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 2 Early 10 GE Tests CERN & SLAC

3. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 3 Throughput Measurements uUDP Throughput with udpmon uSend a controlled stream of UDP frames spaced at regular intervals n bytes Number of packets Wait time time  Zero stats OK done ●●● Get remote statistics Send statistics: No. received No. lost + loss pattern No. out-of-order CPU load & no. int 1-way delay Send data frames at regular intervals ●●● Time to send Time to receive Inter-packet time (Histogram) Signal end of test OK done Time Sender Receiver

4. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 4 PCI Bus & Gigabit Ethernet Activity uPCI Activity uLogic Analyzer with PCI Probe cards in sending PC PCI Probe cards in receiving PC CPU mem chipset NIC CPU mem NIC chipset Logic Analyser Display PCI bus Possible Bottlenecks

5. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 5 10 Gigabit Ethernet: UDP Throughput u1500 byte MTU gives ~ 2 Gbit/s uUsed 16144 byte MTU max user length 16080 uDataTAG Supermicro PCs uDual 2.2 GHz Xenon CPU FSB 400 MHz uPCI-X mmrbc 512 bytes uwire rate throughput of 2.9 Gbit/s uCERN OpenLab HP Itanium PCs uDual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz uPCI-X mmrbc 4096 bytes uwire rate of 5.7 Gbit/s uSLAC Dell PCs giving a uDual 3.0 GHz Xenon CPU FSB 533 MHz uPCI-X mmrbc 4096 bytes uwire rate of 5.4 Gbit/s

6. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 6 10 Gigabit Ethernet: Tuning PCI-X u16080 byte packets every 200 µs uIntel PRO/10GbE LR Adapter uPCI-X bus occupancy vs mmrbc Measured times Times based on PCI-X times from the logic analyser Expected throughput ~7 Gbit/s Measured 5.7 Gbit/s mmrbc 1024 bytes mmrbc 2048 bytes mmrbc 4096 bytes 5.7Gbit/s mmrbc 512 bytes CSR Access PCI-X Sequence Data Transfer Interrupt & CSR Update

7. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 7 Manchester 10 GE Lab

8. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 8 uSuperMicro X5DPE-G2 uDual 2.4 GHz Xeon u 533 MHz Front side bus u 6 PCI PCI-X slots u 4 independent PCI buses 64 bit 66 MHz PCI 100 MHz PCI-X 133 MHz PCI-X u Dual Gigabit Ethernet u UDMA/100 bus master/EIDE channels data transfer rates of 100 MB/sec burst “Server Quality” Motherboards

9. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 9 “Server Quality” Motherboards u Boston/Supermicro H8DAR u Two Dual Core Opterons u 200 MHz DDR Memory Theory BW: 6.4Gbit u HyperTransport u 2 independent PCI buses 133 MHz PCI-X u 2 Gigabit Ethernet u SATA u ( PCI-e )

10. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 10 10 Gigabit Ethernet: iperf TCP Intel Results uX5DPE-G2 Supermicro PCs B2B uDual 2.2 GHz Xeon CPU FSB 533 MHz uXFrame II NIC uPCI-X mmrbc 512 bytes u1500 byte MTU u2.5 Mbyte TCP buffer size uIperf rate throughput of 2.33 Gbit/s uPCI-X mmrbc 512 bytes u9000 byte MTU uIperf rate of 3.92 Gbit/s uPCI-X mmrbc 4096 bytes u9000 byte MTU uIperf rate of 3.94 Gbit/s

11. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 11 10 Gigabit Ethernet: UDP Intel Results uX5DPE-G2 Supermicro PCs B2B uDual 2.2 GHz Xeon CPU FSB 533 MHz uXFrame II NIC uPCI-X mmrbc 4096 bytes uLow rates uLarge packet loss u???

12. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 12 PCI-X Signals from SC2005

13. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 13 10 Gigabit Ethernet: TCP Data transfer on PCI-X uSun V20z 1.8GHz to 2.6 GHz Dual Opterons uConnect via 6509 uXFrame II NIC uPCI-X mmrbc 4096 bytes 66 MHz uTwo 9000 byte packets b2b uAve Rate 2.87 Gbit/s uBurst of packets length 646.8 us uGap between bursts 343 us u2 Interrupts / burst CSR Access Data Transfer

14. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 14 10 Gigabit Ethernet: UDP Data transfer on PCI-X uSun V20z 1.8GHz to 2.6 GHz Dual Opterons uConnect via 6509 uXFrame II NIC uPCI-X mmrbc 2048 bytes 66 MHz uOne 8000 byte packets 2.8us for CSRs 24.2 us data transfer effective rate 2.6 Gbit/s u2000 byte packet wait 0us ~200ms pauses u8000 byte packet wait 0us ~15ms between data blocks CSR Access 2.8us Data Transfer

15. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 15 Disk 2 Disk tests Building on SC2004 work

16. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 16 SC2004 Disk-Disk bbftp ubbftp file transfer program uses TCP/IP uUKLight: Path:- London-Chicago-London; PCs:- Supermicro +3Ware RAID0 uMTU 1500 bytes; Socket size 22 Mbytes; rtt 177ms; SACK off uMove a 2 GByte file uWeb100 plots: uStandard TCP uAverage 825 Mbit/s u(bbcp: 670 Mbit/s) uScalable TCP uAverage 875 Mbit/s u(bbcp: 701 Mbit/s ~4.5s of overhead) uDisk-TCP-Disk at 1Gbit/s

17. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 17 Network & Disk Interactions (Network-Disk sub-system interactions) uHosts: Supermicro X5DPE-G2 motherboards dual 2.8 GHz Zeon CPUs with 512 k byte cache and 1 M byte memory 3Ware 8506-8 controller on 133 MHz PCI-X bus configured as RAID0 six 74.3 GByte Western Digital Raptor WD740 SATA disks 64k byte stripe size uMeasure memory to RAID0 transfer rates with & without UDP traffic Disk write 1735 Mbit/s Disk write + 1500 MTU UDP 1218 Mbit/s Drop of 30% Disk write + 9000 MTU UDP 1400 Mbit/s Drop of 19% % CPU kernel mode

18. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 18 Any Questions?

19. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 19 Backup Slides

20. CALICE UCL, 20 Feb 2006, R. Hughes-Jones Manchester 20 More Information Some URLs 1 uUKLight web site: http://www.uklight.ac.uk uMB-NG project web site: http://www.mb-ng.net/ uDataTAG project web site: http://www.datatag.org/ uUDPmon / TCPmon kit + writeup: http://www.hep.man.ac.uk/~rich/net uMotherboard and NIC Tests: http://www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt & http://datatag.web.cern.ch/datatag/pfldnet2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 http:// www.hep.man.ac.uk/~rich/ uTCP tuning information may be found at: http://www.ncne.nlanr.net/documentation/faq/performance.html & http://www.psc.edu/networking/perf_tune.html uTCP stack comparisons: “Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 uPFLDnet http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/ uDante PERT http://www.geant2.net/server/show/nav.00d00h002