1. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 1 VLBI Data Transfer Tests Recent and Current Work. Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” www.hep.man.ac.uk/~rich/

2. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 2 Outline uThroughput Tests on Mark5s TCP Memory-2-memory tests CPU Load tests uData delay on a TCP link – How suitable is TCP? 4th Year MPhys Project Stephen Kershaw & James Keenan The effect of distance uThroughput on the 630Mbit JB-JIVE UKLight Link TCP Performance

3. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 3 uWhy were Jodrell to JIVE VLBI data transfers not able to do 512 Mbit even on UKLight ? uWhy can Onsala Mk5 achieve 512 Mbps to JIVE Mk5 ? Onsala can even high rates transatlantic – iGrid2005 SC|05? Identical Mk5 hardware to JBO Same kernel and drivers Longer links uHint given as the general Network load increased: Normally Onsala – JIVE iperf TCP ~900-950 Mbit/s VLBI OK at 512 Mbit Sometimes Onsala – JIVE iperf TCP ~750 Mbit/s VLBI not OK at 512 Mbit Is it the network ? Jodrell’s VLBI Mark5 Problem

4. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 4 VLBI Network Topology

5. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 5 VLBI Network Topology

6. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 6 uStandard Mark5 PCs 1.2GHz PIII uEnd host iperf TCP flow memory-to-memory only 960 Mbit/s with rtt 1 ms JBO - Manchester Falls to 770 Mbit/s when rtt 15 ms JBO - JIVE uJBO - Manchester 94.7% kernel mode idle 1.5 % u JBO - JIVE 96.3% kernel mode idle 0.05 % uNo Loss No Timeouts u200* more TCPPureACK seen for JBO-Manchester uTCPHPACKs about the same uHelp with meanings please TCP Tests Jodrell’s Mark5

7. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 7 TCP Throughput & CPU Load uMeasure iperf TCP throughput and CPU load uRun CPU intensive task with different priority (nice High number = low priority) uJBO – Manchester 1.2 GHz PIII TCP Throughput falls as priority increases % Kernel mode drops and %nice increases ad priority increases CPU mode shares with %nice No Loss No Timeouts uJBO – Manchester Asus NCCH-DL 2.8 GHz Xeon TCP Throughput constant as priority increases % Kernel and %nice constant No Loss No Timeouts Onsala has a Faster Clock !

8. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 8 TCP Throughput while reading SuperStor uMeasure iperf TCP throughput while reading data from disk to memory uReading SuperStor from disk to memory only 1.48 Gbit/s uReading SuperStor with iperf 1.15 Gbit/s uIperf TCP rate 420 Mbit/s u15 ms SS read spacing ~1Gbit/s to memory uCorresponding CPU load

9. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 9 TCP Delay and VLBI Transfers Manchester 4th Year MPhys Project by Stephen Kershaw & James Keenan

10. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 10 VLBI Application Protocol u VLBI data is Constant Bit Rate u tcpdelay instrumented TCP program emulates sending CBR Data. Records relative 1-way delay Data1 ●●● Timestamp1 Time TCP & Network Receiver Timestamp2 Sender Data2 Timestamp4 Timestamp5 Data4 Timestamp3 Data3 Packet loss RTT Time Sender Receiver ACK Segment time on wire = bits in segment/BW uRemember Bandwidth*Delay Product BDP = RTT*BW

11. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 11 Send time – 10000 packets Send time sec 1 sec Check the Send Time u10,000 Messages uMessage size: 1448 B uWait time: 0 uTCP buffer 64k uSlope 0.44 ms/message uExpect 42 messages/rtt ~0.6ms/message Message number

12. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 12 Message 102 Message 76 About 25 us One rtt 100 ms Send time sec 26 messages Send Time Detail Message number

13. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 13 1 way delay – 10000 packets 1 way delay 100 ms Message number 1-Way Delay u10,000 Messages uMessage size: 1448 B uWait time: 0 uTCP buffer 64k

14. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 14 Message number = 1.5 x RTT = 1 x RTT 26 ms ≠ 0.5 x RTT 1 way delay 100 ms 1-Way Delay Detail u10,000 Messages uMessage size: 1448 B uWait time: 0 uTCP buffer 64k uWhy not 1 rtt? uWhy does it vary? uEffect of “send time delay” TCP slow start?

15. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 15 Message 102 Message 76 100 ms Send time sec 26 messages Comparison of Send Time & 1-way delay Message number

16. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 16 1 way delay μs Packet number 1 way delay – 10000 packets Packet 1214 1575 packets ~ 5.5 x RTT

17. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 17 u10,000 Messages uMessage size: 724 Bytes uWait times: 20, 25, 30, 35, 40, 45 μs uTCP buffer 64k 1 way delay 100 ms Message number 1-Way Delay 724 byte msg

18. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 18 Packet number 1-Way Delay 724 bytes Detail u10,000 Messages uMessage size: 724 Bytes uWait times: 20, 25, 30, 35, 40, 45 μs uTCP buffer 64k uRegular cycle of ~125 packets 1 way delay 100 ms

19. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 19 uRoute: Man-ukl-ams-prod-man uRtt 27ms u10,000 Messages uMessage size: 1448 Bytes uWait times: 0 μs uDBP = 3.4MByte uTCP buffer 10MByte 1-Way Delay 1448 byte msg 50 ms Message number uWeb100 plot uStarts after 5.6 Sec due to Clock Sync. u~400 pkts/10ms uRate similar to iperf

20. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 20 5 ms Message number uRoute: LAN gig8-gig1 uPing 188us u10,000 Messages uMessage size: 1448 Bytes uWait times: 0 μs uDrop 1 in 1000 1-Way Delay with packet drop 800 us 28 ms

21. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 21 TCP on the 630 Mbit Link Jodrell – UKLight – JIVE

22. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 22 TCP Throughput on 630 Mbit UKLight uManchester gig7 – JBO 606 u4 Mbyte TCP buffer test 0 Dup ACKs seen Other Reductions test 1 test 2

23. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 23 Any Questions?

24. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 24 More Information Some URLs 1 uUKLight web site: http://www.uklight.ac.uk uMB-NG project web site: http://www.mb-ng.net/ uDataTAG project web site: http://www.datatag.org/ uUDPmon / TCPmon kit + writeup: http://www.hep.man.ac.uk/~rich/net uMotherboard and NIC Tests: http://www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt & http://datatag.web.cern.ch/datatag/pfldnet2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 http:// www.hep.man.ac.uk/~rich/ uTCP tuning information may be found at: http://www.ncne.nlanr.net/documentation/faq/performance.html & http://www.psc.edu/networking/perf_tune.html uTCP stack comparisons: “Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 uPFLDnet http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/ uDante PERT http://www.geant2.net/server/show/nav.00d00h002

25. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 25 uLectures, tutorials etc. on TCP/IP: www.nv.cc.va.us/home/joney/tcp_ip.htm www.cs.pdx.edu/~jrb/tcpip.lectures.html www.raleigh.ibm.com/cgi-bin/bookmgr/BOOKS/EZ306200/CCONTENTS www.cisco.com/univercd/cc/td/doc/product/iaabu/centri4/user/scf4ap1.htm www.cis.ohio-state.edu/htbin/rfc/rfc1180.html www.jbmelectronics.com/tcp.htm uEncylopaedia http://www.freesoft.org/CIE/index.htm uTCP/IP Resources www.private.org.il/tcpip_rl.html uUnderstanding IP addresses http://www.3com.com/solutions/en_US/ncs/501302.html uConfiguring TCP (RFC 1122) ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt uAssigned protocols, ports etc (RFC 1010) http://www.es.net/pub/rfcs/rfc1010.txt & /etc/protocols http://www.es.net/pub/rfcs/rfc1010.txt More Information Some URLs 2

26. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 26 Backup Slides

27. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 27 uUDP/IP packets sent between back-to-back systems Processed in a similar manner to TCP/IP Not subject to flow control & congestion avoidance algorithms Used UDPmon test program uLatency uRound trip times measured using Request-Response UDP frames uLatency as a function of frame size Slope is given by: Mem-mem copy(s) + pci + Gig Ethernet + pci + mem-mem copy(s) Intercept indicates: processing times + HW latencies uHistograms of ‘singleton’ measurements uTells us about: Behavior of the IP stack The way the HW operates Interrupt coalescence Latency Measurements

28. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 28 Throughput Measurements uUDP Throughput uSend a controlled stream of UDP frames spaced at regular intervals n bytes Number of packets Wait time time  Zero stats OK done ●●● Get remote statistics Send statistics: No. received No. lost + loss pattern No. out-of-order CPU load & no. int 1-way delay Send data frames at regular intervals ●●● Time to send Time to receive Inter-packet time (Histogram) Signal end of test OK done Time Sender Receiver

29. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 29 PCI Bus & Gigabit Ethernet Activity uPCI Activity uLogic Analyzer with PCI Probe cards in sending PC Gigabit Ethernet Fiber Probe Card PCI Probe cards in receiving PC Gigabit Ethernet Probe CPU mem chipset NIC CPU mem NIC chipset Logic Analyser Display PCI bus Possible Bottlenecks

30. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 30 u SuperMicro P4DP8-2G (P4DP6) uDual Xeon u 400/522 MHz Front side bus u 6 PCI PCI-X slots u 4 independent PCI buses 64 bit 66 MHz PCI 100 MHz PCI-X 133 MHz PCI-X u Dual Gigabit Ethernet u Adaptec AIC-7899W dual channel SCSI u UDMA/100 bus master/EIDE channels data transfer rates of 100 MB/sec burst “Server Quality” Motherboards

31. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 31 “Server Quality” Motherboards u Boston/Supermicro H8DAR u Two Dual Core Opterons u 200 MHz DDR Memory Theory BW: 6.4Gbit u HyperTransport u 2 independent PCI buses 133 MHz PCI-X u 2 Gigabit Ethernet u SATA u ( PCI-e )

32. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 32 Network switch limits behaviour uEnd2end UDP packets from udpmon Only 700 Mbit/s throughput Lots of packet loss Packet loss distribution shows throughput limited

33. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 33 10 Gigabit Ethernet: UDP Throughput u1500 byte MTU gives ~ 2 Gbit/s uUsed 16144 byte MTU max user length 16080 uDataTAG Supermicro PCs uDual 2.2 GHz Xenon CPU FSB 400 MHz uPCI-X mmrbc 512 bytes uwire rate throughput of 2.9 Gbit/s uCERN OpenLab HP Itanium PCs uDual 1.0 GHz 64 bit Itanium CPU FSB 400 MHz uPCI-X mmrbc 4096 bytes uwire rate of 5.7 Gbit/s uSLAC Dell PCs giving a uDual 3.0 GHz Xenon CPU FSB 533 MHz uPCI-X mmrbc 4096 bytes uwire rate of 5.4 Gbit/s

34. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 34 10 Gigabit Ethernet: Tuning PCI-X u16080 byte packets every 200 µs uIntel PRO/10GbE LR Adapter uPCI-X bus occupancy vs mmrbc Measured times Times based on PCI-X times from the logic analyser Expected throughput ~7 Gbit/s Measured 5.7 Gbit/s mmrbc 1024 bytes mmrbc 2048 bytes mmrbc 4096 bytes 5.7Gbit/s mmrbc 512 bytes CSR Access PCI-X Sequence Data Transfer Interrupt & CSR Update

35. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 35 Tests on the UKLight switched light-path Manchester : Dwingeloo u Throughput as a function of inter- packet spacing (2.4 GHz dual Xeon machines) u Packet loss for small packet size uMaximum size packets can reach full line rates with no loss, and there was no re-ordering (plot not shown).

36. ESLEA VLBI Bits&Bytes Workshop, 4-5 May 2006, R. Hughes-Jones Manchester 36 UKLight using Mk5 recording terminals